All Lessons

Importance-Weighted Expectation Estimation

Importance weighting estimates an expectation under one distribution using samples from another. This lesson derives the estimator, support condition, self-normalization, effective sample size, and log-weight stabilization.

AI Narration Press play to listen
0  / 6 paragraphs
Click any paragraph to jump · Scroll freely without breaking narration

Suppose the target is $\mu=\mathbb{E}_{x\sim p}[f(x)]$, but samples are available from proposal distribution $q$. Multiply and divide the integral by $q(x)$ to obtain $$\mu=\mathbb{E}_{x\sim q}\left[\frac{p(x)}{q(x)}f(x)\right].$$ The weight $w(x)=p(x)/q(x)$ corrects the sampling distribution.

With independent samples $x_i\sim q$, the estimator $$\hat\mu=\frac1N\sum_{i=1}^{N}w(x_i)f(x_i)$$ is unbiased when normalized densities are known and the required expectation exists. The essential support condition is $q(x)>0$ wherever $p(x)|f(x)|>0$. Missing support creates information that no finite weight can recover.

Unbiased does not mean low variance. If $q$ rarely samples regions where $p(x)|f(x)|$ is large, a few observations receive enormous weights and dominate the estimate. A useful proposal covers all important target regions and allocates more probability where the magnitude of the integrand is large.

When only unnormalized target density is known, self-normalized estimation uses $$\tilde\mu=\frac{\sum_iw_if(x_i)}{\sum_iw_i}.$$ It is generally biased at finite $N$ but consistent under suitable conditions. Normalization reduces sensitivity to an unknown constant and often improves practical stability.

Weight concentration can be summarized by effective sample size $$N_{eff}=\frac{(\sum_iw_i)^2}{\sum_iw_i^2}.$$ It ranges from near one to $N$ for nonnegative weights. This is a diagnostic rather than a guarantee, but a tiny value warns that the nominal sample count greatly overstates the estimator's information.

Compute log weights $\ell_i=\log p(x_i)-\log q(x_i)$ and subtract their maximum before exponentiation to avoid overflow. Report maximum normalized weight, effective sample size, and repeated-run uncertainty. Clipping weights lowers variance but changes the estimand through bias, so the threshold and resulting sensitivity must be disclosed.