Learned Optimizers Challenge Hand-Tuned Adam

The pitch is compelling, but the gap between “can learn an optimizer” and “can replace Adam” is still mostly an engineering wall, not a conceptual one. The article does a good job showing why meta-optimization is elegant on paper and brutally constrained in practice.

• –Full unrolling quickly becomes expensive because training the optimizer through long trajectories pulls in second-order effects like Hessians.
• –Truncation makes the math tractable, but it biases the learned optimizer toward short-term wins instead of true long-run convergence.
• –Learned optimizers are specialized, amortized policies over a task distribution, not universal drop-in replacements for hand-built optimizers.
• –Generalization can break when the target geometry changes materially, so architecture and activation shifts remain a hard boundary.
• –For AI researchers, the value here is the framing: optimization itself can be learned, but the practical ceiling is still set by compute, stability, and specialization.