What is Attention?

Attention is the core operation of the Transformer: for each token, the model computes how much 'attention' to pay to every other token, then blends their information accordingly. The famous formula — softmax(QK^T/√d)V — is just weighted averaging with learned weights.

Multi-head attention runs this process many times in parallel with different learned projections, letting the model track grammar, coreference, and long-range dependencies simultaneously.

Why it matters: attention is what gives Transformers their memory of context, but its cost grows quadratically with sequence length — the fundamental reason long context is expensive, and why efficient-attention research (FlashAttention, linear attention) is so active.