Affine Divergence reframes activation updates

Hot take: this reads less like “yet another normaliser” and more like a geometry critique of how we train deep nets. If the theory holds up broadly, the interesting part is not scale invariance itself, but the idea that we have been stepping in the wrong activation-space direction all along.

• –The paper’s strongest claim is mechanistic: activation updates can be misaligned even when parameter updates are mathematically correct.
• –The new affine-like layer is notable because it keeps degrees of freedom instead of behaving like a standard normaliser, which makes it a cleaner architectural alternative for MLPs.
• –PatchNorm is the more exploratory idea, but it matters because it extends the same correction logic to convolution, where existing normalisation tricks are often bolted on rather than derived.
• –The reported batch-size effect is a useful falsifiable prediction: if larger batches hurt divergence-correcting layers, that is a sharp signal the mechanism is real, not just an optimization artifact.
• –If these results generalize, they could shift how we think about why LayerNorm, RMSNorm, and BatchNorm help, from “stabilize scale” to “repair update geometry.”