Source author record

Saleh Soltan

Saleh Soltan appears in the imported research catalog. Authorship, coauthor and topic links are available while profile ownership is still unclaimed.

ResearcherUnclaimed source record

Computation and Language Machine Learning Systems and Control Artificial Intelligence Computational Complexity Data Structures and Algorithms Discrete Mathematics math.CO physics.soc-ph

Catalog footprint

What is connected

6works

9topics

4close collaborators

Actions

Connect this record

Open graph Browse works

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

preprint2022arXiv

Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems

We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B, their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system. Though we train using 70% spoken-form data, our teacher models perform comparably to XLM-R and mT5 when evaluated on the written-form Cross-lingual Natural Language Inference (XNLI) corpus. We perform a second stage of pretraining on our teacher models using in-domain data from our system, improving error rates by 3.86% relative for intent classification and 7.01% relative for slot filling. We find that even a 170M-parameter model distilled from our Stage 2 teacher model has 2.88% better intent classification and 7.69% better slot filling error rates when compared to the 2.3B-parameter teacher trained only on public data (Stage 1), emphasizing the importance of in-domain data for pretraining. When evaluated offline using labeled NLU data, our 17M-parameter Stage 2 distilled model outperforms both XLM-R Base (85M params) and DistillBERT (42M params) by 4.23% to 6.14%, respectively. Finally, we present results from a full virtual assistant experimentation platform, where we find that models trained using our pretraining and distillation pipeline outperform models distilled from 85M-parameter teachers by 3.74%-4.91% on an automatic measurement of full-system user dissatisfaction.

preprint2022arXiv

AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.

preprint2016arXiv

A study of cascading failures in real and synthetic power grid topologies using DC power flows

Using the linearized DC power flow model, we study cascading failures and their spatial and temporal properties in the US Western Interconnect (USWI) power grid. We also introduce the preferential Degree And Distance Attachment (DADA) model, with similar degree distributions, resistances, and currents to the USWI. We investigate the behavior of both grids resulting from the failure of a single line. We find that the DADA model and the USWI model react very similarly to that failure, and that their blackout characteristics resemble each other. In many cases, the failure of a single line can cause cascading failures, which impact the entire grid. We characterize the resilience of the grid by three parameters, the most important of which is tolerance $α$, which is the ratio of the maximal load a line can carry to its initial load. We characterize a blackout by its yield, which we define as the ratio of the final to the initial consumed currents. We find that if $α\leq2$, the probability of a large blackout occurring is very small. By contrast, in a broad range of $1 < α < 2$, the initial failure of a single line can result, with a high probability, in cascading failures leading to a massive blackout with final yield less than 80%. The yield has a bimodal distribution typical of a first-order transition, i.e., the failure of a randomly selected line leads either to an insignificant current reduction or to a major blackout. We find that there is a latent period in the development of major blackouts during which few lines are overloaded, and the yield remains high. The duration of this latent period is proportional to the tolerance. The existence of the latent period suggests that intervention during early time steps of a cascade can significantly reduce the risk of a major blackout.

preprint2016arXiv

Doubly Balanced Connected Graph Partitioning

We introduce and study the Doubly Balanced Connected graph Partitioning (DBCP) problem: Let $G=(V,E)$ be a connected graph with a weight (supply/demand) function $p:V\rightarrow \{-1,+1\}$ satisfying $p(V)=\sum_{j\in V} p(j)=0$. The objective is to partition $G$ into $(V_1,V_2)$ such that $G[V_1]$ and $G[V_2]$ are connected, $|p(V_1)|,|p(V_2)|\leq c_p$, and $\max\{\frac{|V_1|}{|V_2|},\frac{|V_2|}{|V_1|}\}\leq c_s$, for some constants $c_p$ and $c_s$. When $G$ is 2-connected, we show that a solution with $c_p=1$ and $c_s=3$ always exists and can be found in polynomial time. Moreover, when $G$ is 3-connected, we show that there is always a `perfect' solution (a partition with $p(V_1)=p(V_2)=0$ and $|V_1|=|V_2|$, if $|V|\equiv 0 (\mathrm{mod}~4)$), and it can be found in polynomial time. Our techniques can be extended, with similar results, to the case in which the weights are arbitrary (not necessarily $\pm 1$), and to the case that $p(V)\neq 0$ and the excess supply/demand should be split evenly. They also apply to the problem of partitioning a graph with two types of nodes into two large connected subgraphs that preserve approximately the proportion of the two types.

preprint2015arXiv

Generation of Synthetic Spatially Embedded Power Grid Networks

The development of algorithms for enhancing the resilience and efficiency of the power grid requires performance evaluation with real topologies of power transmission networks. However, due to security reasons, such topologies and particularly the locations of the substations and the lines are usually not publicly available. Therefore, we study the structural properties of the North American grids and present an algorithm for generating synthetic spatially embedded networks with similar properties to a given grid. The algorithm uses the Gaussian Mixture Model (GMM) for density estimation of the node positions and generates a set of nodes with similar spatial distribution to the nodes in a given network. Then, it uses two procedures, which are inspired by the historical evolution of the grids, to connect the nodes. The algorithm has several tunable parameters that allow generating grids similar to any given grid. Particularly, we apply it to the Western Interconnection (WI) and to grids that operate under the SERC Reliability Corporation (SERC) and the Florida Reliability Coordinating Council (FRCC), and show that it generates grids with similar structural and spatial properties to these grids. To the best of our knowledge, this is the first attempt to consider the spatial distribution of the nodes and lines and its importance in generating synthetic power grids.

preprint2014arXiv

Cascading Failures in Power Grids - Analysis and Algorithms

This paper focuses on cascading line failures in the transmission system of the power grid. Recent large-scale power outages demonstrated the limitations of percolation- and epid- emic-based tools in modeling cascades. Hence, we study cascades by using computational tools and a linearized power flow model. We first obtain results regarding the Moore-Penrose pseudo-inverse of the power grid admittance matrix. Based on these results, we study the impact of a single line failure on the flows on other lines. We also illustrate via simulation the impact of the distance and resistance distance on the flow increase following a failure, and discuss the difference from the epidemic models. We then study the cascade properties, considering metrics such as the distance between failures and the fraction of demand (load) satisfied after the cascade (yield). We use the pseudo-inverse of admittance matrix to develop an efficient algorithm to identify the cascading failure evolution, which can be a building block for cascade mitigation. Finally, we show that finding the set of lines whose removal has the most significant impact (under various metrics) is NP-Hard and introduce a simple heuristic for the minimum yield problem. Overall, the results demonstrate that using the resistance distance and the pseudo-inverse of admittance matrix provides important insights and can support the development of efficient algorithms.