Full self-attention compares every sequence position with every other position. Doubling sequence length creates roughly four times as many pairwise scores. This quadratic score matrix often becomes the memory bottleneck for long contexts.
Computation also grows with model width and projection size, but the sequence-squared term dominates at long lengths. Padding short examples to the longest example wastes work, so grouping similar lengths and packing sequences can improve efficiency.
Optimized kernels can compute exact attention without storing the entire score matrix at once. They reduce memory traffic and intermediate storage while preserving the mathematical result. Kernel efficiency is different from changing which tokens may attend.
Local or sparse attention changes the pattern of allowed connections. It can lower cost by focusing on windows or selected global tokens, but distant information may require several layers to travel. Approximate methods trade exact weights for speed or memory.
During autoregressive generation, a key-value cache avoids recomputing earlier projections, but cache memory grows with sequence length and layers. Grouped key-value heads and reduced precision can shrink that cache with possible quality tradeoffs.
Benchmark peak memory, latency, throughput, and task quality across realistic lengths, not one short input. Confirm numerical agreement when replacing an exact kernel. An efficient method is useful only if its accuracy and hardware behavior match the application.