DrivingBench benchmarks frontier LLMs driving real car

The transition of frontier LLMs from conversational and code-generation agents into closed-loop physical actuators marks a major milestone for embodied reasoning, proving that raw multimodal inference combined with standard tool-use protocols like MCP can pilot real-world machinery—though latency and spatial perception remain formidable bottlenecks.

• –Physical closed-loop control via MCP: Rather than relying on specialized end-to-end autonomous driving models, general-purpose LLMs interacted directly with CAN-bus hardware using simple MCP tool calls (`observe`, `set_motion`, `stop_now`), proving high-level reasoning models can actuate real hardware.
• –In-context error correction: Models demonstrated genuine in-context learning between runs; after failing the first run by misjudging track boundaries, GPT-6 Astra adapted its steering angles (utilizing 100% lock on 20/24 commands) and speed (capping at 0.8 m/s) to complete the course on attempt 2.
• –Inference latency as a physical bottleneck: LLM reasoning times (5–13 second turnaround per turn) meant the vehicle had to move at low speeds; models that failed to overlap asynchronous motion commands spent most of their runs idling while waiting for inference to complete.
• –Perception and alignment friction: Models struggled with spatial scaling (misjudging vehicle width against obstacles) and cone color semantics, while also initially triggering safety refusals against operating a real vehicle until prompted with a sandbox MCP identifier.