After training, generation is an autoregressive loop. Tokenize a prompt, run the transformer, choose one next token from the final logits, append it, and repeat until a stopping rule fires. Every generated token becomes context for the next decision.
Greedy decoding always chooses the highest-scoring token and is reproducible but can become repetitive. Sampling draws from the probability distribution and creates variety. Temperature sharpens or flattens that distribution before sampling.
Top-k sampling keeps only a fixed number of likely tokens. Nucleus sampling keeps the smallest set whose cumulative probability reaches a chosen threshold. Both remove an unstable tail of unlikely choices while leaving more flexibility than greedy decoding.
A key-value cache stores attention projections from earlier positions. Without it, every step recomputes the entire prefix. With it, the model processes only the new token while attending to cached history, greatly reducing repeated work.
Generation must define end tokens, maximum length, and application-specific stop sequences. Batch members may finish at different times. Prompts plus generated tokens must remain within the model's context window, including hidden formatting tokens.
Compare decoding settings on a fixed prompt set. Measure latency, tokens per second, repetition, completion length, and task quality. Keep sampling seeds when reproducing bugs, and distinguish a decoding problem from a model or prompt problem.