Claw-Eval debuts real-world agent benchmark

Claw-Eval looks like a serious attempt to move agent evaluation out of toy prompts and into messy, end-to-end work. If the benchmark stays reproducible and trustworthy, it could become a useful routing layer for deciding when small models are good enough and when frontier models are still worth the spend.

• –104 tasks across 15 services makes this feel closer to operational agent work than a narrow coding eval.
• –Pass^3 is the right instinct: it rewards consistency, not accidental success on a single run.
• –Human verification is a strong signal, but the benchmark will only matter if others can reproduce and extend it cleanly.
• –The biggest downstream use is model orchestration: cheap models for routine steps, frontier models only for hard or orchestration-heavy tasks.
• –Because it includes breakdowns by task and model, it could be especially useful for building a “meta-MoE” router instead of just chasing leaderboard scores.