All Lessons

Expectation, Variance, and Covariance

Expectation describes average behavior, variance measures uncertainty around it, and covariance captures linear co-movement. This lesson derives their identities and connects them to batches, features, and estimators.

AI Narration Press play to listen
0  / 6 paragraphs
Click any paragraph to jump · Scroll freely without breaking narration

For discrete random variable $X$ with probabilities $p(x)$, expectation is $\mathbb{E}[X]=\sum_x xp(x)$. For a continuous variable with density $p(x)$, it is $\mathbb{E}[X]=\int xp(x)\,dx$. Expectation is a probability-weighted average and need not be a value that the variable can actually take.

Expectation is linear: $$\mathbb{E}[aX+bY+c]=a\mathbb{E}[X]+b\mathbb{E}[Y]+c.$$ This identity does not require independence. It explains why the expected mini-batch mean equals the population mean when examples are sampled from the target distribution, even though individual batches fluctuate.

Variance is $$\operatorname{Var}(X)=\mathbb{E}[(X-\mu)^2]=\mathbb{E}[X^2]-\mu^2,$$ where $\mu=\mathbb{E}[X]$. The second form is algebraically useful, while the centered form clarifies meaning. Variance has squared units, and standard deviation $\sqrt{\operatorname{Var}(X)}$ returns to the original units.

Covariance between $X$ and $Y$ is $$\operatorname{Cov}(X,Y)=\mathbb{E}[(X-\mu_X)(Y-\mu_Y)].$$ Positive covariance means above-average values tend to occur together; negative covariance means one tends to be above average when the other is below. Zero covariance rules out linear co-movement but does not generally imply independence.

For random vector $x\in\mathbb{R}^d$, covariance matrix is $\Sigma=\mathbb{E}[(x-\mu)(x-\mu)^T]$. Diagonal entries are feature variances and off-diagonal entries are pairwise covariances. For any vector $v$, $v^T\Sigma v=\operatorname{Var}(v^Tx)\geq0$, so every covariance matrix is positive semidefinite.

Finite data replaces expectations with sample estimates. Report sample count and uncertainty because a variance estimate from a small or dependent sample can be unstable. In machine learning, distinguish variation across examples, across mini-batches, across training runs, and across model predictions; these are different random quantities and answer different questions.