Google’s Carbon Eval Shows OSWorld Method Split

The issue is less that Google ran three trials than that the aggregation rule is easy to miss and materially affects leaderboard comparisons.

• –OSWorld uses an 82-task offline subset, with Carbon and Argon scores maxed over three runs.
• –Best-of-three reporting can reward retries and inflate results versus a one-shot average.
• –Google’s methodology should clearly label this as best-of-3 or pass@3.
• –Competitor scores come from published evaluations with potentially different harnesses and aggregation rules.
• –Developers should demand per-run scores, uncertainty estimates, and matched evaluation protocols before treating rankings as definitive.