Shifted Key GDN trims params, keeps pace

The interesting part is not just the parameter cut; it is that the inductive bias seems to fit linear-attention-style memory better than softmax attention. In other words, GDN may be getting more mileage from state geometry and token-to-token chaining than from learned query/key transforms.

• –The reported gain is small but meaningful: better or equal loss with fewer parameters and faster convergence suggests Q/K projections were partly redundant at this scale.
• –The effect not transferring to softmax attention is the key clue; exact key lookup and normalized retrieval likely depend more on explicit query/key separation there.
• –The shifted-key setup bakes in a strong local continuity prior, which may help linear memory blocks because the hidden state already carries enough context to serve as both selector and address.
• –The results are still narrow: one dataset, undertrained runs, and a specific architecture family, so this is a useful architectural signal rather than proof of a general rule.
• –If it holds up at larger scales, it argues for simpler linear-attention blocks that spend capacity on memory dynamics instead of projection layers.