AutoBE benchmark elevates harness over scale

My take: this is plausible in constrained production workflows, but it is not a general verdict on “local vs frontier” models.

• –Structured function-calling can erase a lot of variance by forcing the model into a narrow, well-typed action space.
• –When the harness validates ASTs, schemas, and compilation, the benchmark measures orchestration discipline as much as raw model intelligence.
• –Tight score clustering is a sign that the eval may be bottlenecked by fixture design, not that model differences disappeared.
• –The four-project setup and strong compliance incentives mean the results likely overstate generalization to messy, ambiguous, real-world backend work.
• –In production, the same pattern usually holds only when the task is decomposed into explicit schemas, compiler checks, and deterministic post-processing.