Warp Benchmarks Expose Open-Weights Cost Trap

Open weights are no longer an automatic cost win; workload-specific measurement matters more than model ideology. Warp’s approach makes model selection an engineering optimization loop rather than a one-time leaderboard decision. Replaying a team’s own tasks produces more actionable results than relying solely on public benchmarks. Cost must be measured alongside correctness, quality, latency, and agent efficiency. Token usage, retries, context length, and harness behavior can erase open-weight pricing advantages. Warp previously reported cutting internal cost per PR from $80 to $30 through targeted benchmarking and routing. Per-task routing is likely more effective than choosing one default model for every workflow.