Abstract

This paper studies convergence of empirical measures smoothed by a Gaussian kernel. Specifically, consider approximating P∗NσP\ast\mathcal{N}_σ, for Nσ≜N(0,σ2Id)\mathcal{N}_σ\triangleq\mathcal{N}(0,σ^2 \mathrm{I}_d), by P^n∗Nσ\hat{P}_n\ast\mathcal{N}_σ, where P^n\hat{P}_n is the empirical measure, under different statistical distances. The convergence is examined in terms of the Wasserstein distance, total variation (TV), Kullback-Leibler (KL) divergence, and χ2χ^2-divergence. We show that the approximation error under the TV distance and 1-Wasserstein distance (W1\mathsf{W}_1) converges at rate eO(d)n−12e^{O(d)}n^{-\frac{1}{2}} in remarkable contrast to a typical n−1dn^{-\frac{1}{d}} rate for unsmoothed W1\mathsf{W}_1 (and d≥3d\ge 3). For the KL divergence, squared 2-Wasserstein distance (W22\mathsf{W}_2^2), and χ2χ^2-divergence, the convergence rate is eO(d)n−1e^{O(d)}n^{-1}, but only if PP achieves finite input-output χ2χ^2 mutual information across the additive white Gaussian noise channel. If the latter condition is not met, the rate changes to ω(n−1)ω(n^{-1}) for the KL divergence and W22\mathsf{W}_2^2, while the χ2χ^2-divergence becomes infinite - a curious dichotomy. As a main application we consider estimating the differential entropy h(P∗Nσ)h(P\ast\mathcal{N}_σ) in the high-dimensional regime. The distribution PP is unknown but nn i.i.d samples from it are available. We first show that any good estimator of h(P∗Nσ)h(P\ast\mathcal{N}_σ) must have sample complexity that is exponential in dd. Using the empirical approximation results we then show that the absolute-error risk of the plug-in estimator converges at the parametric rate eO(d)n−12e^{O(d)}n^{-\frac{1}{2}}, thus establishing the minimax rate-optimality of the plug-in. Numerical results that demonstrate a significant empirical superiority of the plug-in approach to general-purpose differential entropy estimators are provided.

Explore this paper’s authors, topics and related work.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

Reviews 0

Write a reviewWrite

No reviews yet.

Discussion 0

Add a commentWrite

No comments yet.