An intuition for physicists: information gain from experiments
How much one has learned from an experiment is quantifiable by the information gain, also known as the Kullback-Leibler divergence. The narrowing of the posterior parameter distribution $P(θ|D)$ compared with the prior parameter distribution $π(θ)$, is quantified in units of bits, as: $ D_{\mathrm{KL}}(P|π)=\int\log_{2}\left(\frac{P(θ|D)}{π(θ)}\right)\,P(θ|D)\,dθ$. This research note gives an intuition what one bit of information gain means. It corresponds to a Gaussian shrinking its standard deviation by a factor of three.