LHTB benchmark tests long-horizon terminal AI agents
Long-Horizon-Terminal-Bench (LHTB) is a robust evaluation benchmark containing 46 tasks across nine domains designed to test AI agents on complex, long-running terminal tasks. Instead of binary pass/fail metrics, LHTB uses dense reward-based grading to measure incremental progress, revealing that frontier models frequently fail due to timeouts during these hours-long evaluations.
While frontier models excel at short-term instructions, LHTB exposes their current inability to maintain state and persist productively over long horizons.
* Decomposing tasks into graded subtasks with dense rewards represents a major step forward for agent evaluation, reducing the noise of binary metrics.
* The finding that 79% of failures are due to timeouts suggests that context-window drift and execution inefficiency, rather than raw reasoning capability, are the primary bottlenecks.
* With task runs averaging 9.9 million tokens, testing on LHTB is extremely resource-intensive, highlighting the need for more efficient agent scaffolding architectures.
DISCOVERED
4h ago
2026-07-13
PUBLISHED
4h ago
2026-07-13
RELEVANCE
AUTHOR
_akhaliq