Macroscope launches MacroscopeBench for AI code review

Evaluating AI code reviewers on synthetic defects has always been security theater; testing models blindly on historical commits that actually broke production is the only realistic way to reveal the trade-off between bug detection and pull request noise.

* Realistic evaluation harness: By testing models on raw diffs at the point of introduction without indicating whether defects exist, the benchmark accurately mirrors true production code review conditions.

* Anti-hallucination and noise controls: The inclusion of clean control commits and dual LLM judges penalizes subjective nitpicks and phantom issues, reflecting the high cost of false positives in engineering workflows.

* Trade-offs made visible: Measuring latency and API inference costs alongside precision and recall exposes whether frontier models justify their expense over lighter, faster alternatives for continuous CI/CD reviews.

* LLM judge dependency: Evaluating findings with LLM judges rather than automated execution or test suites leaves some room for evaluator inconsistency when judging subtle domain-specific bugs.