Hospital-bench tests agents on hospital operations
Zavo's Hospital-bench evaluates AI agents inside a simulated health system featuring 13 connected applications, 131 operational tools, and 48 administrative tasks like prior authorizations and claims management. Across 20 frontier models tested, no model passed more than 17 of the 48 tasks, underscoring critical reliability deficits in multi-step enterprise workflows.
Hospital-bench delivers a sobering reality check for enterprise automation: agent reliability degrades sharply when navigating asynchronous software, contradictory records, and strict regulatory deadlines. While foundation models show strong reasoning in controlled tests, messy operational coordination remains largely unsolved.
- –Low task completion ceiling: Top scorer GPT-6.2 Astra averaged 62.4% but passed only 14 of 48 tasks, while Claude Opus 5.5 led passes at 17 of 48; 14 tasks defeated all 20 tested models.
- –Deterministic evaluation: The benchmark relies on 1,678 programmatic database state checks and 162 safety rules rather than subjective LLM evaluators, penalizing wrong actions and zeroing scores for harmful outcomes like voiding paid claims.
- –Simulated real-world chaos: Agents must navigate 13 disparate apps and 149 unsearchable policy documents containing conflicting records, stale caches, and decoys designed to penalize reliance on pre-trained assumptions.
- –Fragile execution patterns: Common model failures included skipping mandated Medicaid verification steps, prematurely closing external tickets, and waiting out execution clocks without submitting required handoffs.
- –Enterprise deployment reality: Results indicate that frontier agents are currently suitable only for drafting and queue triage, requiring human coordinators to oversee state-altering healthcare operations.
DISCOVERED
3h ago
2026-11-20
PUBLISHED
4h ago
2026-11-20
RELEVANCE
AUTHOR
omarsar0