Priors adds real-world T3 Code telemetry
AI benchmark platform Priors updated its 'Time per task' metric to default to real-world developer telemetry from T3 Code rather than synthetic speed estimates. Early production data reveals Claude models getting faster in multi-turn coding agent workflows, while Gemini and OpenAI models skew slower.
Synthetic tokens-per-second benchmarks consistently fail to capture how frontier models actually perform inside autonomous agent loops. By ingesting live telemetry from T3 Code, Priors exposes the true wall-clock execution costs that developers face in production.
- –Production vs synthetic divergence: While Gemini often leads in raw output token streaming speeds, complex reasoning overhead and tool-calling roundtrips make it noticeably slower in real agentic coding sessions.
- –Claude maintains coding efficiency: Claude models show faster real-world task completion times, likely driven by optimized prompt caching and leaner multi-step tool execution.
- –Granular telemetry pipeline: The new metric pulls daily aggregate data from T3 Code's ClickHouse dashboard across 30 days of completed turns, segmented by model and reasoning effort.
- –Sample-size transparency: Hovering over data points now reveals the user and turn counts behind each figure, filtering out thin samples to prevent single-session skew.
- –Practical guidance for builders: Teams orchestrating coding agents can now evaluate model latencies based on actual developer usage rather than artificial vendor benchmarks.
DISCOVERED
6h ago
2026-11-16
PUBLISHED
7h ago
2026-11-16
RELEVANCE
AUTHOR
AdamHoltererer