RedstoneBench exposes Grok’s redstone weakness

This is a compelling agent benchmark because success requires producing hardware that actually works, not merely generating plausible code.

• –Covers doors, mechanics, logic, memory, and precise timing tasks.
• –Scores compactness and execution speed, making bloated or sluggish solutions less competitive.
• –Tick-by-tick vanilla verification avoids the subjectivity of model-based judging.
• –Grok’s weak showing suggests limitations in spatial planning, state tracking, or iterative debugging.
• –The leaderboard remains preliminary: Claude has not yet been tested, and some tasks lack human baselines.