Many-species ecological fluctuations as a jump process from the brink of extinction
Highly-diverse ecosystems exhibit a broad distribution of population sizes and species turnover, where species at high and low abundances are exchanged over time. We show that these two features generically emerge in the fluctuating phase of many-variable model ecosystems with disordered species interactions, when species are supported by migration from outside the system at a small rate. We show that these and other phenomena can be understood through the existence of a scaling regime in the limit of small migration, in which large fluctuations and long timescales emerge. We construct an exact analytical theory for this asymptotic regime, that provides scaling predictions on timescales and abundance distributions that are verified exactly in simulations. In this regime, a clear separation emerges between rare and abundant species at any given time, despite species moving back and forth between the rare and abundant subsets. The number of abundant species is found to lie strictly below a well-known stability bound, maintaining the system away from marginality. At the same time, other measures of diversity, which also include some of the rare species, go above this bound. In the asymptotic limit where the migration rate goes to zero, trajectories of individual species abundances are described by non-Markovian jump-diffusion processes, which proceeds as follows: A rare species remains so for some time, then experiences a jump in population sizes after which it becomes abundant (a species turnover event) and later sees its population size gradually decreasing again until rare, due to the competition with other species. The asymmetry of abundance trajectories under time-reversal is maintained at small but finite migration rate. These features may serve as fingerprints of endogenous fluctuations in highly-diverse ecosystems.