A policy can collapse too early onto actions that looked good from limited experience. Entropy regularization adds a preference for distributions that retain uncertainty, giving learning more opportunities to test alternatives. For a discrete policy, the entropy at state $s$ is $$\mathcal{H}(\pi(\cdot\mid s))=-\sum_a\pi(a\mid s)\log\pi(a\mid s).$$
The entropy-regularized objective is $$J(\pi)=\mathbb{E}_\pi\left[\sum_{t=0}^{\infty}\gamma^t\left(r_{t+1}+\alpha\mathcal{H}(\pi(\cdot\mid s_t))\right)\right].$$ The temperature $\alpha\geq0$ sets the exchange rate between reward and randomness. At zero, the objective reduces to ordinary discounted reward; larger values favor broader action distributions.
For a fixed state and action scores $Q(s,a)$, maximizing expected score plus entropy yields a soft policy proportional to $$\pi(a\mid s)\propto\exp(Q(s,a)/\alpha).$$ Small temperature concentrates probability on the largest score, while large temperature flattens differences. This connects entropy regularization to smooth action selection rather than an arbitrary noise process.
Temperature can be learned by targeting an entropy level. A common objective adjusts $\alpha$ according to the gap between observed log probability and target entropy. The target must match the action space and policy parameterization. A target that is too high can prevent exploitation; one that is too low can allow premature collapse.
For continuous actions, differential entropy depends on units and coordinate transformations, so its absolute value is less universal than discrete entropy. Squashed or bounded distributions also require the transformation's log-density correction. Omitting that term produces incorrect policy gradients and misleading entropy measurements.
High entropy is not the same as effective or safe exploration. A policy can be random in irrelevant action dimensions while never reaching informative states. Track entropy by state region, action saturation, return, and coverage of meaningful outcomes. Evaluate with a separate deployment policy if stochastic training behavior is not appropriate at inference time.