OPEN LINK ↗
// 1h agoRESEARCH PAPER
Meta's AutoRefine Calibrates Code Judges
Meta’s AutoRefine paper presents a prompt-rewriting loop that uses noisy workflow outcomes—tests, merges, and abandons—to recalibrate LLM code judges, while abstaining when outcomes cannot be explained by the judge’s inputs. On held-out diffs from a deployed coding agent, agreement rose from Scott’s π 0.16 to 0.51. ([paper](https://arxiv.org/abs/2611.09291))
// ANALYSIS
The key insight is treating judge calibration as production maintenance, not a one-time prompt exercise. However, merge and abandon outcomes remain imperfect proxies for code quality, making abstention and drift monitoring essential.
- –Workflow outcomes provide cheap supervision, while the teacher filters labels it cannot justify.
- –Injected label noise fell from 25% to 20% after teacher-based abstention.
- –The recalibrated judge improved on current diffs but lost badly on legacy diffs, proving agent evolution creates calibration drift.
- –Hidden-test failures remain invisible when the judge lacks sufficient evidence; better inputs may matter more than better prompts.
- –The study is promising but small, internal, and based on one deployed agent and shared model families.
// TAGS
autorefinellmevaluationcoding-agenttestingprompt-engineeringresearch
DISCOVERED
1h ago
2026-11-10
PUBLISHED
1h ago
2026-11-10
RELEVANCE
9/ 10
AUTHOR
omarsar0