BrowserOS-Style Tests Lose Their Bite

Single-file tasks are still good smoke tests, but they’re increasingly a floor, not a ceiling. The real benchmark now is whether an agent can keep state, navigate ambiguity, and survive feedback loops across a whole codebase.

• –Single-file prompts mostly test local pattern matching, syntax repair, and one-shot completion.
• –Stronger evals should include repo-wide dependencies, hidden tests, and iterative debugging with logs and failing CI.
• –Agentic coding needs tool-use benchmarks: search, edit, run, inspect, retry, and recover from bad assumptions.
• –Private benchmarks are most useful when they mirror real team workflows, not leaderboard-friendly toy problems.