Pi Agent Tools Need Benchmarks

This is the right question, because agent tool tweaks are easy to fool yourself on: a few smoother runs can feel like a breakthrough even when the long-tail behavior is unchanged. The only defensible approach is to evaluate against a fixed task set with telemetry, then compare against the old tooling.

• –Build a locked benchmark from real sessions: same repo snapshot, same prompt, same success criteria, repeated over time
• –Measure hard metrics, not just "feels faster": task success, wall-clock time, token usage, tool-call count, retries, repeated reads, and test pass rate
• –Include negative and trap cases where the custom tool should not help, so you can spot overfitting or prompt nudges
• –Re-run across multiple models, quantizations, and inference settings, since a tool improvement on one Qwen setup may not generalize
• –Keep a regression set for the exact failure modes here, like log dumping and blind rereads, because those are the behaviors most likely to come back