Moonshot AI debuts Attention Residuals

Moonshot AI's Kimi team introduces Attention Residuals (AttnRes), a novel architecture that replaces fixed residual connections with a depth-wise attention mechanism. By allowing layers to selectively retrieve information from all preceding representations, AttnRes achieves a 1.25x compute advantage and effectively mitigates signal dilution in deep Transformer models.