A causal language model learns by predicting the next token at every position. One text sequence supplies many training examples at once: each prefix predicts the token immediately following it. This simple objective can teach syntax, facts, and patterns present in the data.
The model outputs one score, called a logit, for every vocabulary token at each position. Softmax converts logits to probabilities. Cross-entropy penalizes low probability on the actual next token, and training averages that penalty across valid positions.
Teacher forcing supplies the true earlier tokens while predicting later ones. A causal mask prevents each position from seeing future tokens, otherwise the model could copy the answer. Padding positions must also be excluded from both attention and loss.
Suppose the input is 'the sky is' and the target continuation is 'blue'. The model scores every vocabulary item at the final position. If blue receives low probability, the loss is large and gradients adjust all contributing transformer layers.
Training loss alone can fall through memorization. Keep validation text separate and remove near duplicates across splits. Data order, sequence packing, truncation, and document boundaries all affect what relationships the model can learn.
Monitor training and validation loss, token accuracy, gradient norm, learning rate, throughput, and numerical precision. Sudden loss spikes may indicate bad batches or unstable optimization. Evaluate generated samples too, because a correct average loss can hide repetitive decoding behavior.