Replica tops frontier models on research replication

This is a more meaningful agent benchmark than another static knowledge test: successful replication requires turning ambiguous scientific prose into working evidence.

• –Held-out papers reduce the risk of overfitting to familiar benchmarks or training data
• –Replication exposes omitted details, brittle assumptions, and implementation gaps that ordinary coding tasks miss
• –A 27B model winning here suggests task design, training signals, and agent scaffolding may matter as much as parameter count
• –Converting research workflows into RL environments could create a scalable path toward stronger autonomous science agents
• –The result still needs methodology, run counts, cost, and error analysis before the headline should be treated as definitive