SRE-Bench exposes AI’s binary-analysis gap

SRE-Bench is a much-needed reality check for cybersecurity agents: strong source-code performance does not automatically transfer to compiled software.

• –19 private programs and 44 anti-analysis mechanisms reduce memorization and shortcutting
• –The strongest initial model fully solved only 31.5% of instances
• –Tasks span malware, firmware, proprietary software, and other realistic reverse-engineering scenarios
• –Agents’ relative insensitivity to optimization and static linking suggests pattern matching remains a major weakness
• –Deterministic grading makes the benchmark more meaningful than evaluations based on plausible explanations