Harness-IF exposes coding agents' default bias

AGENTS.md guidance is only as reliable as the agent’s willingness to depart from its defaults, making rule-level, execution-based evaluation far more useful than simply checking task completion.

• –Evaluates 256 rules across 60 realistic multi-turn coding tasks and five instruction surfaces
• –Introduces Against-Prior Accuracy to separate genuine instruction following from behavior the model would have produced anyway
• –Every tested model performed worse on rules opposing its defaults, with gaps ranging from 3.6 to 7.4 points
• –Most failures were omissions of required actions, challenging the assumption that agents mainly violate prohibitions
• –The benchmark’s judge-heavy scoring means its broad patterns are more convincing than small leaderboard rank differences