BZPEERPapers, people, evidence and research pathsDiscover
Research tools
Network
Opportunities
Account
Source author record
Persi Diaconis appears in the imported research catalog. Authorship, coauthor and topic links are available while profile ownership is still unclaimed.
Catalog footprint
Research graph
Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.
BZPEER is loading the nearby papers, people, topics and institutions for this page.
Published work
The Rado graph, also known as the random graph $G(\infty, p)$, is a classical limit object for finite graphs. We study natural ball walks as a way of understanding the geometry of this graph. For the walk started at $i$, we show that order $\log_2^*i$ steps are sufficient, and for infinitely many $i$, necessary for convergence to stationarity. The proof involves an application of Hardy's inequality for trees.
Let $G$ be a finite group. Let $H, K$ be subgroups of $G$ and $H \backslash G / K$ the double coset space. Let $Q$ be a probability on $G$ which is constant on conjugacy classes ($Q(s^{-1} t s) = Q(t)$). The random walk driven by $Q$ on $G$ projects to a Markov chain on $H \backslash G /K$. This allows analysis of the lumped chain using the representation theory of $G$. Examples include coagulation-fragmentation processes and natural Markov chains on contingency tables. Our main example projects the random transvections walk on $GL_n(q)$ onto a Markov chain on $S_n$ via the Bruhat decomposition. The chain on $S_n$ has a Mallows stationary distribution and interesting mixing time behavior. The projection illuminates the combinatorics of Gaussian elimination. Along the way, we give a representation of the sum of transvections in the Hecke algebra of double cosets. Some extensions and examples of double coset Markov chains with $G$ a compact group are discussed.
The two of us have shared a fascination with James Victor Uspensky's 1937 textbook $Introduction \, to \, Mathematical \, Probability$ ever since our graduate student days: it contains many interesting results not found in other books on the same subject in the English language, together with many non-trivial examples, all clearly stated with careful proofs. We present some of Uspensky's gems to a modern audience hoping to tempt others to read Uspensky for themselves, as well as report on a few of the other mathematical topics he also wrote about (for example, his book on number theory contains early results about perfect shuffles). Uspensky led an interesting life: a member of the Russian Academy of Sciences, he spoke at the 1924 International Congress of Mathematicians in Toronto before leaving Russia in 1929 and coming to the US and Stanford. Comparatively little has been written about him in English; the second half of this paper attempts to remedy this.
Consider two independent Erdős-Rényi $G(N,1/2)$ graphs. We show that with probability tending to $1$ as $N\to\infty$, the largest induced isomorphic subgraph has size either $\lfloor x_N-\varepsilon_N\rfloor$ or $\lfloor x_N+\varepsilon_N \rfloor$, where $x_N=4\log_2 N -2 \log_2 \log_2 N - 2\log_2(4/e)+1$ and $\varepsilon_N = (4\log_2 N)^{-1/2}$. Using similar techniques, we also show that if $Γ_1$ and $Γ_2$ are independent $G(n,1/2)$ and $G(N,1/2)$ random graphs, then $Γ_2$ contains an isomorphic copy of $Γ_1$ as an induced subgraph with high probability if $n\le \lfloor y_N - \varepsilon_N \rfloor$ and does not contain an isomorphic copy of $Γ_1$ as an induced subgraph with high probability if $n>\lfloor y_N+\varepsilon_N \rfloor$, where $y_N=2\log_2 N+1$ and $\varepsilon_N$ is as above.
This paper makes three contributions to estimating the number of perfect matching in bipartite graphs. First, we prove that the popular sequential importance sampling algorithm works in polynomial time for dense bipartite graphs. More carefully, our algorithm gives a $(1\pmε)$-approximation for the number of perfect matchings of a $λ$-dense bipartite graph, using $O(n^{\frac{1-2λ}λε^{-2}})$ samples. With size $n$ on each side and for $\frac{1}{2}>λ>0$, a $λ$-dense bipartite graph has all degrees greater than $(λ+\frac{1}{2})n$. Second, practical applications of the algorithm require many calls to matching algorithms. A novel preprocessing step is provided which makes significant improvements. Third, three applications are provided. The first is for counting Latin squares, the second is a practical way of computing the greedy algorithm for a card-guessing game with feedback, and the third is for stochastic block models. In all three examples, sequential importance sampling allows treating practical problems of reasonably large sizes.
We study a natural Markov chain on $\{0,1,\cdots,n\}$ with eigenvectors the Hahn polynomials. This explicit diagonalization makes it possible to get sharp rates of convergence to stationarity. The process, the Burnside process, is a special case of the celebrated `Swendsen-Wang' or `data augmentation' algorithm. The description involves the beta-binomial distribution and Mallows model on permutations. It introduces a useful generalization of the Burnside process.
Let $H$ and $K$ be subgroups of a finite group $G$. Pick $g \in G$ uniformly at random. We study the distribution induced on double cosets. Three examples are treated in detail: 1) $H = K = $ the Borel subgroup in $GL_n(\mathbb{F}_q)$. This leads to new theorems for Mallows measure on permutations and new insights into the LU matrix factorization. 2) The double cosets of the hyperoctahedral group inside $S_{2n}$, which leads to new applications of the Ewens's sampling formula of mathematical genetics. 3) Finally, if $H$ and $K$ are parabolic subgroups of $S_n$, the double cosets are `contingency tables', studied by statisticians for the past 100 years.
We show that the convergence of finite state space Markov chains to stationarity can often be considerably speeded up by alternating every step of the chain with a deterministic move. Under fairly general conditions, we show that not only do such schemes exist, they are numerous.
This paper does three things: It proves a central limit theorem for novel permutation statistics (for example, the number of descents plus the number of descents in the inverse). It provides a clear illustration of a new approach to proving central limit theorems more generally. It gives us an opportunity to acknowledge the work of our teacher and friend B. V. Rao.
The purpose of this article is to put forward the claim that Hurwitz's paper "Uber die Erzeugung der Invarianten durch Integration." [Gott. Nachrichten (1897), 71-90] should be regarded as the origin of random matrix theory in mathematics. Here Hurwitz introduced and developed the notion of an invariant measure for the matrix groups $SO(N)$ and $U(N)$. He also specified a calculus from which the explicit form of these measures could be computed in terms of an appropriate parametrisation - Hurwitz chose to use Euler angles. This enabled him to define and compute invariant group integrals over $SO(N)$ and $U(N)$. His main result can be interpreted probabilistically: the Euler angles of a uniformly distributed matrix are independent with beta distributions (and conversely). We use this interpretation to give some new probability results. How Hurwitz's ideas and methods show themselves in the subsequent work of Weyl, Dyson and others on foundational studies in random matrix theory is detailed.
We explore the link between combinatorics and probability generated by the question "What does a random parking function look like?" This gives rise to novel probabilistic interpretations of some elegant, known generating functions. It leads to new combinatorics: how many parking functions begin with $i$? We classify features (e.g., the full descent pattern) of parking functions that have exactly the same distribution among parking functions as among all functions. Finally, we develop the link between parking functions and Brownian excursion theory to give examples where the two ensembles differ.
Let H(n) be the group of 3x3 uni-uppertriangular matrices with entries in Z/nZ, the integers mod n. We show that the simple random walk converges to the uniform distribution in order n^2 steps. The argument uses Fourier analysis and is surprisingly challenging. It introduces novel techniques for bounding the spectrum which are useful for a variety of walks on a variety of groups.
We prove the conjectured limiting normality for the number of crossings of a uniformly chosen set partition of [n] = {1,2,...,n}. The arguments use a novel stochastic representation and are also used to prove central limit theorems for the dimension index and the number of levels.
Consider a finite absorbing Markov generator, irreducible on the non-absorbing states. Perron-Frobenius theory ensures the existence of a corresponding positive eigenvector $φ$. The goal of the paper is to give bounds on the amplitude $\max φ/\minφ$. Two approaches are proposed: one using a path method and the other one, restricted to the reversible situation, based on spectral estimates. The latter approach is extended to denumerable birth and death processes absorbing at 0 for which infinity is an entrance boundary. The interest of estimating the ratio is the reduction of the quantitative study of convergence to quasi-stationarity to the convergence to equilibrium of related ergodic processes, as seen in [7].
In analyzing a simple random walk on the Heisenberg group we encounter the problem of bounding the extreme eigenvalues of an $n\times n$ matrix of the form $M=C+D$ where $C$ is a circulant and $D$ a diagonal matrix. The discrete Schrödinger operators are an interesting special case. The Weyl and Horn bounds are not useful here. This paper develops three different approaches to getting good bounds. The first uses the geometry of the eigenspaces of $C$ and $D$, applying a discrete version of the uncertainty principle. The second shows that, in a useful limit, the matrix $M$ tends to the harmonic oscillator on $L^2(\mathbb{R})$ and the known eigenstructure can be transferred back. The third approach is purely probabilistic, extending $M$ to an absorbing Markov chain and using hitting time arguments to bound the Dirichlet eigenvalues. The approaches allow generalization to other walks on other groups.
Orthogonal polynomials for the multinomial distribution m(x, p) of N balls dropped into d boxes (box i has probability p(i)) are called multivariate Krawtchouk polynomials. This paper gives an introduction to their properties, collections of natural Markov chains which they explicitly diagonalize and associated bivariate multinomial distributions.
This article gives a rigorous analysis of the fluctuations of the Bose-Einstein condensate for a system of non-interacting bosons in an arbitrary potential, assuming that the system is governed by the canonical ensemble. As a result of the analysis, we are able to tell the order of fluctuations of the condensate fraction as well as its limiting distribution upon proper centering and scaling. This yields interesting results. For example, for a system of $n$ bosons in a 3D harmonic trap near the transition temperature, the order of fluctuations of the condensate fraction is $n^{-1/2}$ and the limiting distribution is normal, whereas for the 3D uniform Bose gas, the order of fluctuations is $n^{-1/3}$ and the limiting distribution is an explicit non-normal distribution. For a 2D harmonic trap, the order of fluctuations is $n^{-1/2}(\log n)^{1/2}$, which is larger than $n^{-1/2}$ but the limiting distribution is still normal. All of these results come as easy consequences of a general theorem.
The quantitative long time behavior of absorbing, finite, irreducible Markov processes is considered. Via Doob transforms, it is shown that only the knowledge of the ratio of the values of the underlying first Dirichlet eigenvector is necessary to come back to the well-investigated situation of the convergence to equilibrium of ergodic finite Markov processes. This leads to explicit estimates on the convergence to quasi-stationarity, in particular via functional inequalities. When the process is reversible, the optimal exponential rate consisting of the spectral gap between the two first Dirichlet eigenvalues is recovered. Several simple examples are provided to illustrate the bounds obtained.
We consider new types of perfect shuffles wherein a deck is split in half, one half of the deck is "reversed", and then the cards are interlaced. Flip shuffles are when the reversal comes from flipping the half over so that we also need to account for face-up/face-down configurations while horseshoe shuffles are when the order of the cards are reversed but all cards still face the same direction. We show that these shuffles are closely related to faro shuffling and determine the order of the associated shuffling groups.
In a uniform random permutation Πof [n] := {1,2,...,n}, the set of elements k in [n-1] such that Π(k+1) = Π(k) + 1 has the same distribution as the set of fixed points of Πthat lie in [n-1]. We give three different proofs of this fact using, respectively, an enumeration relying on the inclusion-exclusion principle, the introduction of two different Markov chains to generate uniform random permutations, and the construction of a combinatorial bijection. We also obtain the distribution of the analogous set for circular permutations that consists of those k in [n] such that Π(k+1 mod n) = Π(k) + 1 mod n. This latter random set is just the set of fixed points of the commutator [ρ, Π], where ρis the n-cycle (1,2,...,n). We show for a general permutation ηthat, under weak conditions on the number of fixed points and 2-cycles of η, the total variation distance between the distribution of the number of fixed points of [η,Π] and a Poisson distribution with expected value 1 is small when n is large.
Many casinos routinely use mechanical card shuffling machines. We were asked to evaluate a new product, a shelf shuffler. This leads to new probability, new combinatorics and to some practical advice which was adopted by the manufacturer. The interplay between theory, computing, and real-world application is developed.
Given a group G and a normal subgroup H we study the problem of choosing coset representatives with few carries.
In studying the enumerative theory of super characters' of the group of upper triangular matrices over a finite field we found that the moments (mean, variance and higher moments) of novel statistics on set partitions have simple closed expressions as linear combinations of shifted bell numbers. It is shown here that families of other statistics have similar moments. The coefficients in the linear combinations are polynomials in $n$. This allows exact enumeration of the moments for small $n$ to determine exact formulae for all $n$.
We study the combinatorics of addition using balanced digits, deriving an analog of Holte's "amazing matrix" for carries in usual addition. The eigenvalues of this matrix for base b balanced addition of n numbers are found to be 1,1/b,...,1/b^n, and formulas are given for its left and right eigenvectors. It is shown that the left eigenvectors can be identified with hyperoctahedral Foulkes characters, and that the right eigenvectors can be identified with hyperoctahedral Eulerian idempotents. We also examine the carries that occur when a column of balanced digits is added, showing this process to be determinantal. The transfer matrix method and a serendipitous diagonalization are used to study this determinantal process.
Contributed discussion and rejoinder to "Geodesic Monte Carlo on Embedded Manifolds" (arXiv:1301.6064)
We introduce a method for the theoretical analysis of exponential random graph models. The method is based on a large-deviations approximation to the normalizing constant shown to be consistent using theory developed by Chatterjee and Varadhan [European J. Combin. 32 (2011) 1000-1017]. The theory explains a host of difficulties encountered by applied workers: many distinct models have essentially the same MLE, rendering the problems ``practically'' ill-posed. We give the first rigorous proofs of ``degeneracy'' observed in these models. Here, almost all graphs have essentially no edges or are essentially complete. We supplement recent work of Bhamidi, Bresler and Sly [2008 IEEE 49th Annual IEEE Symposium on Foundations of Computer Science (FOCS) (2008) 803-812 IEEE] showing that for many models, the extra sufficient statistics are useless: most realizations look like the results of a simple Erdős-Rényi model. We also find classes of models where the limiting graphs differ from Erdős-Rényi graphs. A limitation of our approach, inherited from the limitation of graph limit theory, is that it works only for dense graphs.
The operation of squaring (coproduct followed by product) in a combinatorial Hopf algebra is shown to induce a Markov chain in natural bases. Chains constructed in this way include widely studied methods of card shuffling, a natural "rock-breaking" process, and Markov chains on simplicial complexes. Many of these chains can be explictly diagonalized using the primitive elements of the algebra and the combinatorics of the free Lie algebra. For card shuffling, this gives an explicit description of the eigenvectors. For rock-breaking, an explicit description of the quasi-stationary distribution and sharp rates to absorption follow.
This paper offers a personal review of some things we've learned about rates of convergence of Markov chains to their stationary distributions. The main topic is ways of speeding up diffusive behavior. It also points to open problems and how much more there is to do.
Repeated convolution of a probability measure on Z leads to the central limit theorem and other limit theorems. This paper investigates what kinds of results remain without positivity. It reviews theorems due to Schoenberg, Greville, and Thomee which are motivated by applications to data smoothing (Schoenberg and Greville) and finite difference schemes (Thomee). Using Fourier transform arguments, we prove detailed decay bounds for convolution powers of finitely supported complex functions on Z. If M is an hermitian contraction, an estimate for the off-diagonal entries of the powers M^n k of M^k = I -(I-M)^k is obtained. This generalizes the Carne-Varopoulos Markov chain estimate.
If L is a partition of n, the rank of L is the size of the largest part minus the number of parts. Under the uniform distribution on partitions, Bringmann, Mahlburg, and Rhoades showed that the rank statistic has a limiting distribution. We identify the limit as the difference between two independent extreme value distributions and as the distribution of B(T) where B(t) is standard Brownian motion and T is the first time that an independent three-dimensional Brownian motion hits the unit sphere. The same limit holds for the crank.
We consider a general class of discrete, two-dimensional Markov chains modeling the dynamics of a population with two types, without mutation or immigration, and neutral in the sense that type has no influence on each individual's birth or death parameters. We prove that all the eigenvectors of the corresponding transition matrix or infinitesimal generator Π can be expressed as the product of "universal" polynomials of two variables, depending on each type's size but not on the specific transitions of the dynamics, and functions depending only on the total population size. These eigenvectors appear to be Dirichlet eigenvectors for Π on the complement of triangular subdomains, and as a consequence the corresponding eigenvalues are ordered in a specific way. As an application, we study the quasistationary behavior of finite, nearly neutral, two-dimensional Markov chains, absorbed in the sense that 0 is an absorbing state for each component of the process.
We develop algorithms for sampling from a probability distribution on a submanifold embedded in Rn. Applications are given to the evaluation of algorithms in 'Topological Statistics'; to goodness of fit tests in exponential families and to Neyman's smooth test. This article is partially expository, giving an introduction to the tools of geometric measure theory.
John Holte [16] introduced a family of "amazing matrices" which give the transition probabilities of "carries" when adding a list of numbers. It was subsequently shown that these same matrices arise in the combinatorics of the Veronese embedding of commutative algebra [4,6,7] and in the analysis of riffle shuffling [6,7]. We find that the left eigenvectors of these matrices form the Foulkes character table of the symmetric group and the right eigenvectors are the Eulerian idempotents introduced by Loday [20] in work on Hochschild homology. The connections give new closed formulae for Foulkes characters and allow explicit computation of natural correlation functions in the original carries problem.
This paper gives sharp rates of convergence for natural versions of the Metropolis algorithm for sampling from the uniform distribution on a convex polytope. The singular proposal distribution, based on a walk moving locally in one of a fixed, finite set of directions, needs some new tools. We get useful bounds on the spectrum and eigenfunctions using Nash and Weyl-type inequalities. The top eigenvalues of the Markov chain are closely related to the Neuman eigenvalues of the polytope for a novel Laplacian.
Large graphs are sometimes studied through their degree sequences (power law or regular graphs). We study graphs that are uniformly chosen with a given degree sequence. Under mild conditions, it is shown that sequences of such graphs have graph limits in the sense of Lovász and Szegedy with identifiable limits. This allows simple determination of other features such as the number of triangles. The argument proceeds by studying a natural exponential model having the degree sequence as a sufficient statistic. The maximum likelihood estimate (MLE) of the parameters is shown to be unique and consistent with high probability. Thus $n$ parameters can be consistently estimated based on a sample of size one. A fast, provably convergent, algorithm for the MLE is derived. These ingredients combine to prove the graph limit theorem. Along the way, a continuous version of the Erdős--Gallai characterization of degree sequences is derived.
The well-known Gilbert-Shannon-Reeds model for riffle shuffles assumes that the cards are initially cut 'about in half' and then riffled together. We analyze a natural variant where the initial cut is biased. Extending results of Fulman (1998), we show a sharp cutoff in separation and L-infinity distances. This analysis is possible due to the close connection between shuffling and quasisymmetric functions along with some complex analysis of a generating function.
We identify two seemingly disparate structures: supercharacters, a useful way of doing Fourier analysis on the group of unipotent uppertriangular matrices with coefficients in a finite field, and the ring of symmetric functions in noncommuting variables. Each is a Hopf algebra and the two are isomorphic as such. This allows developments in each to be transferred. The identification suggests a rich class of examples for the emerging field of combinatorial Hopf algebras.
The two-parameter Macdonald polynomials are a central object of algebraic combinatorics and representation theory. We give a Markov chain on partitions of k with eigenfunctions the coefficients of the Macdonald polynomials when expanded in the power sum polynomials. The Markov chain has stationary distribution a new two-parameter family of measures on partitions, the inverse of the Macdonald weight (rescaled). The uniform distribution on permutations and the Ewens sampling formula are special cases. The Markov chain is a version of the auxiliary variables algorithm of statistical physics. Properties of the Macdonald polynomials allow a sharp analysis of the running time. In natural cases, a bounded number of steps suffice for arbitrarily large k.
Many seemingly disparate Markov chains are unified when viewed as random walks on the set of chambers of a hyperplane arrangement. These include the Tsetlin library of theoretical computer science and various shuffling schemes. If only selected features of the chains are of interest, then the mixing times may change. We study the behavior of hyperplane walks, viewed on a subarrangement of a hyperplane arrangement. These include many new examples, for instance a random walk on the set of acyclic orientations of a graph. All such walks can be treated in a uniform fashion, yielding diagonalizable matrices with known eigenvalues, stationary distribution and good rates of convergence to stationarity.
Consider the barycentric subdivision which cuts a given triangle along its medians to produce six new triangles. Uniformly choosing one of them and iterating this procedure gives rise to a Markov chain. We show that almost surely, the triangles forming this chain become flatter and flatter in the sense that their isoperimetric values goes to infinity with time. Nevertheless, if the triangles are renormalized through a similitude to have their longest edge equal to $[0,1]\subset\CC$ (with 0 also adjacent to the shortest edge), their aspect does not converge and we identify the limit set of the opposite vertex with the segment [0,1/2]. In addition we prove that the largest angle converges to $π$ in probability. Our approach is probabilistic and these results are deduced from the investigation of a limit iterated random function Markov chain living on the segment [0,1/2]. The stationary distribution of this limit chain is particularly important in our study. In an appendix we present related numerical simulations (not included in the version submitted for publication).
We investigate the properties of uniform doubly stochastic random matrices, that is non-negative matrices conditioned to have their rows and columns sum to 1. The rescaled marginal distributions are shown to converge to exponential distributions and indeed even large sub-matrices of side-length $o(n^{1/2-ε})$ behave like independent exponentials. We determine the limiting empirical distribution of the singular values the the matrix. Finally the mixing time of the associated Markov chains is shown to be exactly 2 with high probability.
We study how many riffle shuffles are required to mix n cards if only certain features of the deck are of interest, e.g. suits disregarded or only the colors of interest. For these features, the number of shuffles drops from 3/2 log_2(n) to log_2(n). We derive closed formulae and an asymptotic `rule of thumb' formula which is remarkably accurate.
We consider a Markov chain on the space of (countable) partitions of the interval [0,1], obtained first by size biased sampling twice (allowing repetitions) and then merging the parts (if the sampled parts are distinct) or splitting the part uniformly (if the same part was sampled twice). We prove a conjecture of Vershik stating that the Poisson-Dirichlet law with parameter theta=1 is the unique invariant distribution for this Markov chain. Our proof uses a combination of probabilistic, combinatoric, and representation-theoretic arguments.
We study a reversible one-dimensional spin system with Bernoulli(p) stationary distribution, in which a site can flip only if the site to its left is in state +1. Such models have been used as simple exemplars of systems exhibiting slow relaxation. We give fairly sharp estimates of the spectral gap as p decreases to zero. The method uses Poincare comparison with a long-range process which is analyzed by probabilistic methods (coupling, supermartingales).