Estimating the number of unseen species: A bird in the hand is worth log⁡n\log n in the bush

preprint2016arXivOpen access

Abstract

Estimating the number of unseen species is an important problem in many scientific endeavors. Its most popular formulation, introduced by Fisher, uses nn samples to predict the number UU of hitherto unseen species that would be observed if t⋅nt\cdot n new samples were collected. Of considerable interest is the largest ratio tt between the number of new and existing samples for which UU can be accurately predicted. In seminal works, Good and Toulmin constructed an intriguing estimator that predicts UU for all t≤1t\le 1, thereby showing that the number of species can be estimated for a population twice as large as that observed. Subsequently Efron and Thisted obtained a modified estimator that empirically predicts UU even for some t>1t>1, but without provable guarantees. We derive a class of estimators that provably\textit{provably} predict UU not just for constant t>1t>1, but all the way up to tt proportional to log⁡n\log n. This shows that the number of species can be estimated for a population log⁡n\log n times larger than that observed, a factor that grows arbitrarily large as nn increases. We also show that this range is the best possible and that the estimators' mean-square error is optimal up to constants for any tt. Our approach yields the first provable guarantee for the Efron-Thisted estimator and, in addition, a variant which achieves stronger theoretical and experimental performance than existing methodologies on a variety of synthetic and real datasets. The estimators we derive are simple linear estimators that are computable in time proportional to nn. The performance guarantees hold uniformly for all distributions, and apply to all four standard sampling models commonly used across various scientific disciplines: multinomial, Poisson, hypergeometric, and Bernoulli product.

Explore this paper’s authors, topics and related work.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

Reviews 0

Write a reviewWrite

No reviews yet.

Discussion 0

Add a commentWrite

No comments yet.