Attention, 2017 → 2026 · 34 mechanisms in launch order

Paying less for attention

Vanilla attention was never wrong. It was expensive. Everything since 2017 is someone looking at the bill and trying to pay less of one line of it. Laid out by the date each idea actually appeared, you can watch the field change its mind about which line hurts most.

Every date links to the paper or release it came from. How they were checked, and the few that could only be pinned down to a month or an upper bound, are in the README.

Start here: the standard one, and its bill

Each token makes a query ("what am I looking for?"), a key ("what do I contain?") and a value ("what I hand over if picked"). Its query is compared with every earlier key, the scores go through a softmax, and the output is the weighted average of the values:

Attention(Q, K, V) = softmax( Q·Kᵀ / √d + causal mask ) · V
Drag the query
Dots are keys (squashed to 2-D); colours are values. The query's direction picks which keys win. Its length sets how decisive the softmax is. The output is the blend of the winners' colours. The √d divisor exists to keep real 128-dimensional queries at a sensible "length".
The bill, as context grows
Compute
score calculations per head per layer (n × n)
Memory
KV cache, Llama-3-8B shape, full multi-head (32 KV heads, 16-bit)
Exactness
to store one layer's n × n score matrix for 32 heads naïvely
Length
positions: will the model even work at this n?
Compute and the score matrix grow with n². The cache grows with n but is paid on every generated token. Positions decide whether you can reach large n at all. These four lines are the colours used on the timeline below.

The timeline

Each dot is a mechanism, placed by the date it first appeared and in the lane of the bill it mainly attacks. Filled dots were on the class list; rings are additions. The band on top names what the field was mostly trying to buy at the time. Years from 2023 on are drawn wider than earlier ones, because that is where most of the mechanisms are; the break on the axis marks the change. Select a dot to jump to its card.

Judge each one for
Show

What the timeline shows that a list doesn't

The field swings between two bills, and positions decide when either comes due. Compute grows with n², memory with n, and the position scheme decides how large n can get before the model breaks. In date order, the attention swings between those lines in a pattern you can't see when mechanisms are grouped by family:

  1. 2017–18 Encode order. Learned, sinusoidal, relative.
  2. 2019–21 Go long by approximating. Fixed sparse, top-k, LSH, low-rank, linear. Exactness is given up to reach long sequences. MQA (2019) arrives here too and is mostly ignored.
  3. 2022 Exact again. FlashAttention makes exact attention fast enough that most approximations quietly disappear.
  4. May 2023 Memory. GQA, because the cache now fills the GPU.
  5. Jun–Sep 2023 Length. Position Interpolation, NTK-aware scaling, YaRN and attention sinks within about 14 weeks.
  6. 2024 Memory back. MLA, while linear attention is quietly rebuilt with the delta rule and gates.
  7. 2025 Go long again, but trainably. NSA, MoBA and DSA learn what to skip; hybrids (MiniMax-01, Qwen3-Next, Kimi Linear) keep a few exact layers. Then a reversal: MiniMax goes back to full attention on the same day Kimi Linear claims a hybrid beats it.
  8. Dec 2025–2026 Positions dropped, memory again. DroPE removes positions after training. DeepSeek-V4 compresses the cache DSA had left untouched.

Ideas arrive years before their bill comes due. Almost nothing in 2025 is new as an idea; what's new is that its line of the bill started to hurt.

  • MQA (Nov 2019) became GQA and a default only in 2023, once long contexts made the cache the problem.
  • Top-k selection (Dec 2019) came back as DSA (Sep 2025), once there was a cheap indexer to do the choosing.
  • The delta rule (Feb 2021) became Gated DeltaNet (Dec 2024) and then 3 of every 4 layers in Qwen3-Next.
  • NoPE (May 2023) became iRoPE (Apr 2025) and then DroPE (Dec 2025).
  • Attention sinks went from an observed quirk (2023) to a learned parameter in gpt-oss (2025) and DeepSeek-V4 (2026).

Every fix moves the bill to another line.

  • A sliding window saves compute, and then needs sinks to stay stable.
  • MLA saves memory, and in exchange costs decode compute and needs a workaround for RoPE.
  • DSA saves compute but keeps the full cache, which is exactly what CSA/HCA went after next.
  • Linear attention saves both, loses recall, and gets hybridised. The hybrids then lose multi-hop reasoning at scale, and one lab retreats.

Exactness keeps winning the argument. Every approximation era has ended with exact attention made cheaper rather than replaced: FlashAttention in 2022, MiniMax M2 in 2025. The sparse designs that survived all keep an exact path: a sliding window of raw tokens, plus the selected tokens at full resolution.

So what comes next?

If the pattern holds, after "memory again" comes length again. But this time the limit isn't compute or memory, since both have been cut hard. It's whether the model can use a million tokens. That points at the position line: position-free inference (DroPE-style), learned rather than fixed positions, and training on genuinely long agent traces. I'd also expect "learned selection over a compressed memory" to spread from DeepSeek to everyone, and the next fight to be about where that compressed memory lives (GPU, CPU or disk). That's a guess, not a date, so treat it as one.