For class probabilities $p_1(x),\ldots,p_K(x)$, predicted label is $\hat y=\arg\max_k p_k(x)$ and confidence is $c(x)=\max_k p_k(x)$. A classifier is confidence-calibrated when examples assigned confidence near $q$ are correct about fraction $q$ of the time. Accuracy and calibration are distinct: a less accurate model can still report more truthful probabilities.
A reliability diagram partitions predictions into bins $B_m$. For each bin, compare average confidence $conf(B_m)$ with empirical accuracy $acc(B_m)$. A common summary is $$ECE=\sum_{m=1}^{M}\frac{|B_m|}{n}|acc(B_m)-conf(B_m)|.$$ This estimate depends on bin edges and sample count, so it should not be treated as an exact population quantity.
Proper probability losses evaluate the full distribution. Log loss is $-\log p_y(x)$ for true class $y$, while squared probability loss is $\sum_k(p_k(x)-\mathbf{1}[y=k])^2$. Both penalize confident errors strongly and are minimized in expectation by reporting the true conditional distribution. They measure more than calibration alone.
A simple post-training adjustment divides logits by positive temperature $T$: $$p_k^{(T)}(x)=\frac{e^{z_k/T}}{\sum_j e^{z_j/T}}.$$ Fit $T$ on a held-out calibration set. This changes confidence without changing class ranking, so it can improve calibration but cannot repair wrong decision boundaries or subgroup errors.
Selective prediction abstains when confidence is below threshold $\tau$. Coverage is $P(c(x)\geq\tau)$, and selective risk is error rate among accepted predictions. Raising $\tau$ usually lowers coverage and may lower risk, producing a risk-coverage curve. The threshold should reflect the downstream cost of errors and deferrals rather than an arbitrary confidence target.
Calibration can break under distribution shift even when held-out in-domain metrics look good. Report results by subgroup, difficulty, and time period; include uncertainty intervals for sparse bins. Monitor both coverage and realized risk after deployment. An abstention system is only safer when deferred cases reach a process capable of handling them.