Paper Lantern open-sources benchmark for RAG coding agents

This benchmark proves that parametric memory isn't enough; giving agents access to recent CS literature is a massive structural advantage.

• –The biggest gains came from tasks requiring modern, post-training techniques published in 2026, which standard baseline models failed to implement.
• –Baseline agents default to basic pre-training priors, while RAG agents dynamically discover and apply advanced techniques like mutation-aware prompting.
• –The results highlight a new failure mode: self-refinement can actually hurt performance when agents second-guess themselves after reading contradictory literature.
• –The fully reproducible eval runs in 10 minutes on a free API key, setting a high bar for transparent agent benchmarking.