RBF Attention Trades Dot Products for Distance

This is a useful proof-of-concept, not a near-term FlashAttention replacement. The interesting part is less the math than the systems lesson: attention kernels, positional encoding, and sink behavior are all coupled to dot-product assumptions.

• –The RBF formulation is effectively dot-product attention plus a key-norm penalty, which makes the geometry more explicit without changing the basic softmax machinery
• –A custom kernel was necessary because fused SDPA paths are optimized around the standard attention contract, not arbitrary score shaping
• –The register-token workaround for attention sinks is a practical reminder that some “bugs” in transformers are actually learned stability mechanisms
• –Dropping RoPE for additive sinusoidal embeddings is a coherent choice once distance, not angle, is the scoring primitive
• –The TinyStories result is interesting but not decisive; this reads more like a researchy systems experiment than a validated production alternative