EAURNNR Proposes Top-K Recurrent Memory

The idea is interesting, but the hardest part is also the most obvious risk: hard top-k routing usually makes training fragile unless you accept approximation tricks and their tradeoffs.

• –The top-k gradient issue is real. STE or soft-to-hard annealing can work as a prototype path, but they often introduce noisy gradients, instability, or a gap between training and inference behavior.
• –The two-timescale memory idea has legs. A fast hidden state plus slow EMA memory is a sensible way to separate local updates from longer-horizon context, and it is more concrete than vague "memory" claims in many recurrent proposals.
• –The main question is capacity, not just complexity. A single persistent vector can help continuity, but it may be too compressed for rich long-context retrieval unless the model learns very disciplined write/read behavior.
• –The architecture’s practical value depends on a parallel training story. Without a scan-friendly or chunked formulation, the theoretical savings may be overshadowed by sequential training bottlenecks.
• –Compared with Mamba, RWKV, and RetNet, the proposal is more selective and more explicitly memory-centric, but it also takes on more optimization risk at once.