BZPEERReading, trust and collaboration for researchDiscover
Workspaces
Network
Opportunities
Account
Researcher profile
Persi Diaconis contributes to research discovery and scholarly infrastructure.
Trust snapshot
Actions
Identity and collaboration
Claiming links this public author record to a researcher profile and unlocks direct collaboration workflows.
Log in to claimDirect collaboration
Claim this author entity first to unlock direct invitations.
Research graph
Inspect adjacent work, topics, institutions and collaborators without jumping out to a separate graph page.
BZPEER is loading the nearby papers, people, topics and institutions for this page.
Published work
The Rado graph, also known as the random graph $G(\infty, p)$, is a classical limit object for finite graphs. We study natural ball walks as a way of understanding the geometry of this graph. For the walk started at $i$, we show that order $\log_2^*i$ steps are sufficient, and for infinitely many $i$, necessary for convergence to stationarity. The proof involves an application of Hardy's inequality for trees.
Let $G$ be a finite group. Let $H, K$ be subgroups of $G$ and $H \backslash G / K$ the double coset space. Let $Q$ be a probability on $G$ which is constant on conjugacy classes ($Q(s^{-1} t s) = Q(t)$). The random walk driven by $Q$ on $G$ projects to a Markov chain on $H \backslash G /K$. This allows analysis of the lumped chain using the representation theory of $G$. Examples include coagulation-fragmentation processes and natural Markov chains on contingency tables. Our main example projects the random transvections walk on $GL_n(q)$ onto a Markov chain on $S_n$ via the Bruhat decomposition. The chain on $S_n$ has a Mallows stationary distribution and interesting mixing time behavior. The projection illuminates the combinatorics of Gaussian elimination. Along the way, we give a representation of the sum of transvections in the Hecke algebra of double cosets. Some extensions and examples of double coset Markov chains with $G$ a compact group are discussed.
The two of us have shared a fascination with James Victor Uspensky's 1937 textbook $Introduction \, to \, Mathematical \, Probability$ ever since our graduate student days: it contains many interesting results not found in other books on the same subject in the English language, together with many non-trivial examples, all clearly stated with careful proofs. We present some of Uspensky's gems to a modern audience hoping to tempt others to read Uspensky for themselves, as well as report on a few of the other mathematical topics he also wrote about (for example, his book on number theory contains early results about perfect shuffles). Uspensky led an interesting life: a member of the Russian Academy of Sciences, he spoke at the 1924 International Congress of Mathematicians in Toronto before leaving Russia in 1929 and coming to the US and Stanford. Comparatively little has been written about him in English; the second half of this paper attempts to remedy this.
Consider two independent Erdős-Rényi $G(N,1/2)$ graphs. We show that with probability tending to $1$ as $N\to\infty$, the largest induced isomorphic subgraph has size either $\lfloor x_N-\varepsilon_N\rfloor$ or $\lfloor x_N+\varepsilon_N \rfloor$, where $x_N=4\log_2 N -2 \log_2 \log_2 N - 2\log_2(4/e)+1$ and $\varepsilon_N = (4\log_2 N)^{-1/2}$. Using similar techniques, we also show that if $Γ_1$ and $Γ_2$ are independent $G(n,1/2)$ and $G(N,1/2)$ random graphs, then $Γ_2$ contains an isomorphic copy of $Γ_1$ as an induced subgraph with high probability if $n\le \lfloor y_N - \varepsilon_N \rfloor$ and does not contain an isomorphic copy of $Γ_1$ as an induced subgraph with high probability if $n>\lfloor y_N+\varepsilon_N \rfloor$, where $y_N=2\log_2 N+1$ and $\varepsilon_N$ is as above.
This paper makes three contributions to estimating the number of perfect matching in bipartite graphs. First, we prove that the popular sequential importance sampling algorithm works in polynomial time for dense bipartite graphs. More carefully, our algorithm gives a $(1\pmε)$-approximation for the number of perfect matchings of a $λ$-dense bipartite graph, using $O(n^{\frac{1-2λ}λε^{-2}})$ samples. With size $n$ on each side and for $\frac{1}{2}>λ>0$, a $λ$-dense bipartite graph has all degrees greater than $(λ+\frac{1}{2})n$. Second, practical applications of the algorithm require many calls to matching algorithms. A novel preprocessing step is provided which makes significant improvements. Third, three applications are provided. The first is for counting Latin squares, the second is a practical way of computing the greedy algorithm for a card-guessing game with feedback, and the third is for stochastic block models. In all three examples, sequential importance sampling allows treating practical problems of reasonably large sizes.
We study a natural Markov chain on $\{0,1,\cdots,n\}$ with eigenvectors the Hahn polynomials. This explicit diagonalization makes it possible to get sharp rates of convergence to stationarity. The process, the Burnside process, is a special case of the celebrated `Swendsen-Wang' or `data augmentation' algorithm. The description involves the beta-binomial distribution and Mallows model on permutations. It introduces a useful generalization of the Burnside process.
Let $H$ and $K$ be subgroups of a finite group $G$. Pick $g \in G$ uniformly at random. We study the distribution induced on double cosets. Three examples are treated in detail: 1) $H = K = $ the Borel subgroup in $GL_n(\mathbb{F}_q)$. This leads to new theorems for Mallows measure on permutations and new insights into the LU matrix factorization. 2) The double cosets of the hyperoctahedral group inside $S_{2n}$, which leads to new applications of the Ewens's sampling formula of mathematical genetics. 3) Finally, if $H$ and $K$ are parabolic subgroups of $S_n$, the double cosets are `contingency tables', studied by statisticians for the past 100 years.
We show that the convergence of finite state space Markov chains to stationarity can often be considerably speeded up by alternating every step of the chain with a deterministic move. Under fairly general conditions, we show that not only do such schemes exist, they are numerous.
Many casinos routinely use mechanical card shuffling machines. We were asked to evaluate a new product, a shelf shuffler. This leads to new probability, new combinatorics and to some practical advice which was adopted by the manufacturer. The interplay between theory, computing, and real-world application is developed.
Given a group G and a normal subgroup H we study the problem of choosing coset representatives with few carries.
In studying the enumerative theory of super characters' of the group of upper triangular matrices over a finite field we found that the moments (mean, variance and higher moments) of novel statistics on set partitions have simple closed expressions as linear combinations of shifted bell numbers. It is shown here that families of other statistics have similar moments. The coefficients in the linear combinations are polynomials in $n$. This allows exact enumeration of the moments for small $n$ to determine exact formulae for all $n$.
We study the combinatorics of addition using balanced digits, deriving an analog of Holte's "amazing matrix" for carries in usual addition. The eigenvalues of this matrix for base b balanced addition of n numbers are found to be 1,1/b,...,1/b^n, and formulas are given for its left and right eigenvectors. It is shown that the left eigenvectors can be identified with hyperoctahedral Foulkes characters, and that the right eigenvectors can be identified with hyperoctahedral Eulerian idempotents. We also examine the carries that occur when a column of balanced digits is added, showing this process to be determinantal. The transfer matrix method and a serendipitous diagonalization are used to study this determinantal process.
We introduce a method for the theoretical analysis of exponential random graph models. The method is based on a large-deviations approximation to the normalizing constant shown to be consistent using theory developed by Chatterjee and Varadhan [European J. Combin. 32 (2011) 1000-1017]. The theory explains a host of difficulties encountered by applied workers: many distinct models have essentially the same MLE, rendering the problems ``practically'' ill-posed. We give the first rigorous proofs of ``degeneracy'' observed in these models. Here, almost all graphs have essentially no edges or are essentially complete. We supplement recent work of Bhamidi, Bresler and Sly [2008 IEEE 49th Annual IEEE Symposium on Foundations of Computer Science (FOCS) (2008) 803-812 IEEE] showing that for many models, the extra sufficient statistics are useless: most realizations look like the results of a simple Erdős-Rényi model. We also find classes of models where the limiting graphs differ from Erdős-Rényi graphs. A limitation of our approach, inherited from the limitation of graph limit theory, is that it works only for dense graphs.
This paper offers a personal review of some things we've learned about rates of convergence of Markov chains to their stationary distributions. The main topic is ways of speeding up diffusive behavior. It also points to open problems and how much more there is to do.
Repeated convolution of a probability measure on Z leads to the central limit theorem and other limit theorems. This paper investigates what kinds of results remain without positivity. It reviews theorems due to Schoenberg, Greville, and Thomee which are motivated by applications to data smoothing (Schoenberg and Greville) and finite difference schemes (Thomee). Using Fourier transform arguments, we prove detailed decay bounds for convolution powers of finitely supported complex functions on Z. If M is an hermitian contraction, an estimate for the off-diagonal entries of the powers M^n k of M^k = I -(I-M)^k is obtained. This generalizes the Carne-Varopoulos Markov chain estimate.
We consider a general class of discrete, two-dimensional Markov chains modeling the dynamics of a population with two types, without mutation or immigration, and neutral in the sense that type has no influence on each individual's birth or death parameters. We prove that all the eigenvectors of the corresponding transition matrix or infinitesimal generator Π can be expressed as the product of "universal" polynomials of two variables, depending on each type's size but not on the specific transitions of the dynamics, and functions depending only on the total population size. These eigenvectors appear to be Dirichlet eigenvectors for Π on the complement of triangular subdomains, and as a consequence the corresponding eigenvalues are ordered in a specific way. As an application, we study the quasistationary behavior of finite, nearly neutral, two-dimensional Markov chains, absorbed in the sense that 0 is an absorbing state for each component of the process.
We develop algorithms for sampling from a probability distribution on a submanifold embedded in Rn. Applications are given to the evaluation of algorithms in 'Topological Statistics'; to goodness of fit tests in exponential families and to Neyman's smooth test. This article is partially expository, giving an introduction to the tools of geometric measure theory.
John Holte [16] introduced a family of "amazing matrices" which give the transition probabilities of "carries" when adding a list of numbers. It was subsequently shown that these same matrices arise in the combinatorics of the Veronese embedding of commutative algebra [4,6,7] and in the analysis of riffle shuffling [6,7]. We find that the left eigenvectors of these matrices form the Foulkes character table of the symmetric group and the right eigenvectors are the Eulerian idempotents introduced by Loday [20] in work on Hochschild homology. The connections give new closed formulae for Foulkes characters and allow explicit computation of natural correlation functions in the original carries problem.
This paper gives sharp rates of convergence for natural versions of the Metropolis algorithm for sampling from the uniform distribution on a convex polytope. The singular proposal distribution, based on a walk moving locally in one of a fixed, finite set of directions, needs some new tools. We get useful bounds on the spectrum and eigenfunctions using Nash and Weyl-type inequalities. The top eigenvalues of the Markov chain are closely related to the Neuman eigenvalues of the polytope for a novel Laplacian.
Large graphs are sometimes studied through their degree sequences (power law or regular graphs). We study graphs that are uniformly chosen with a given degree sequence. Under mild conditions, it is shown that sequences of such graphs have graph limits in the sense of Lovász and Szegedy with identifiable limits. This allows simple determination of other features such as the number of triangles. The argument proceeds by studying a natural exponential model having the degree sequence as a sufficient statistic. The maximum likelihood estimate (MLE) of the parameters is shown to be unique and consistent with high probability. Thus $n$ parameters can be consistently estimated based on a sample of size one. A fast, provably convergent, algorithm for the MLE is derived. These ingredients combine to prove the graph limit theorem. Along the way, a continuous version of the Erdős--Gallai characterization of degree sequences is derived.
The well-known Gilbert-Shannon-Reeds model for riffle shuffles assumes that the cards are initially cut 'about in half' and then riffled together. We analyze a natural variant where the initial cut is biased. Extending results of Fulman (1998), we show a sharp cutoff in separation and L-infinity distances. This analysis is possible due to the close connection between shuffling and quasisymmetric functions along with some complex analysis of a generating function.
We identify two seemingly disparate structures: supercharacters, a useful way of doing Fourier analysis on the group of unipotent uppertriangular matrices with coefficients in a finite field, and the ring of symmetric functions in noncommuting variables. Each is a Hopf algebra and the two are isomorphic as such. This allows developments in each to be transferred. The identification suggests a rich class of examples for the emerging field of combinatorial Hopf algebras.
The two-parameter Macdonald polynomials are a central object of algebraic combinatorics and representation theory. We give a Markov chain on partitions of k with eigenfunctions the coefficients of the Macdonald polynomials when expanded in the power sum polynomials. The Markov chain has stationary distribution a new two-parameter family of measures on partitions, the inverse of the Macdonald weight (rescaled). The uniform distribution on permutations and the Ewens sampling formula are special cases. The Markov chain is a version of the auxiliary variables algorithm of statistical physics. Properties of the Macdonald polynomials allow a sharp analysis of the running time. In natural cases, a bounded number of steps suffice for arbitrarily large k.
Many seemingly disparate Markov chains are unified when viewed as random walks on the set of chambers of a hyperplane arrangement. These include the Tsetlin library of theoretical computer science and various shuffling schemes. If only selected features of the chains are of interest, then the mixing times may change. We study the behavior of hyperplane walks, viewed on a subarrangement of a hyperplane arrangement. These include many new examples, for instance a random walk on the set of acyclic orientations of a graph. All such walks can be treated in a uniform fashion, yielding diagonalizable matrices with known eigenvalues, stationary distribution and good rates of convergence to stationarity.
Consider the barycentric subdivision which cuts a given triangle along its medians to produce six new triangles. Uniformly choosing one of them and iterating this procedure gives rise to a Markov chain. We show that almost surely, the triangles forming this chain become flatter and flatter in the sense that their isoperimetric values goes to infinity with time. Nevertheless, if the triangles are renormalized through a similitude to have their longest edge equal to $[0,1]\subset\CC$ (with 0 also adjacent to the shortest edge), their aspect does not converge and we identify the limit set of the opposite vertex with the segment [0,1/2]. In addition we prove that the largest angle converges to $π$ in probability. Our approach is probabilistic and these results are deduced from the investigation of a limit iterated random function Markov chain living on the segment [0,1/2]. The stationary distribution of this limit chain is particularly important in our study. In an appendix we present related numerical simulations (not included in the version submitted for publication).
We investigate the properties of uniform doubly stochastic random matrices, that is non-negative matrices conditioned to have their rows and columns sum to 1. The rescaled marginal distributions are shown to converge to exponential distributions and indeed even large sub-matrices of side-length $o(n^{1/2-ε})$ behave like independent exponentials. We determine the limiting empirical distribution of the singular values the the matrix. Finally the mixing time of the associated Markov chains is shown to be exactly 2 with high probability.
We study how many riffle shuffles are required to mix n cards if only certain features of the deck are of interest, e.g. suits disregarded or only the colors of interest. For these features, the number of shuffles drops from 3/2 log_2(n) to log_2(n). We derive closed formulae and an asymptotic `rule of thumb' formula which is remarkably accurate.