For a discrete distribution $p$ over outcomes $x$, entropy is $$H(p)=-\sum_xp(x)\log p(x).$$ It is the expected negative log probability of an outcome sampled from $p$. Entropy is zero for a certain outcome and is largest for a uniform distribution over a fixed finite support.
If data follows $p$ but a model reports distribution $q$, cross-entropy is $$H(p,q)=-\sum_xp(x)\log q(x).$$ It measures expected surprise when outcomes from $p$ are encoded or scored using $q$. Assigning very small probability to an event that occurs creates a large penalty.
Relative entropy is $$D(p\Vert q)=\sum_xp(x)\log\frac{p(x)}{q(x)}=H(p,q)-H(p).$$ It is nonnegative and equals zero only when distributions match wherever $p$ has mass. It is asymmetric because expectations are taken under $p$, so reversing the arguments asks a different question.
In supervised classification with one-hot target $y$, per-example cross-entropy is $-\sum_ky_k\log q_k=-\log q_y$. Averaging over data estimates $H(p,q)$. Since data entropy $H(p)$ does not depend on model parameters, minimizing cross-entropy also minimizes relative entropy from the data distribution to the model within the chosen model family.
The log base sets units. Natural logarithms produce nats; base-two logarithms produce bits. Optimization gradients differ only by a constant scale. More importantly, if $q(x)=0$ where $p(x)>0$, relative entropy is infinite. Stable implementations operate on logits with a combined log-softmax operation rather than taking the log of rounded probabilities.
Low cross-entropy does not imply calibrated uncertainty on every subgroup or under distribution shift. Decompose evaluation by class and confidence, inspect rare-event losses, and compare accuracy with proper probability scores. A model can improve top-one decisions while assigning increasingly unrealistic probability to alternatives.