Carbon Medium tops HearingBench leaderboard
Adam Holter updated HearingBench, a multimodal evaluation benchmark testing vision-language models on spectrogram interpretation without audio input. Gemini 4 Carbon Medium clinched a new state of the art over Carbon High, while newly tested models Terra and Hydrogen missed the Pareto frontier.
Testing whether vision models can interpret audio purely by reading spectrograms reveals unexpected inversions in reasoning scaling, where medium thinking effort can actually outperform higher compute tiers. Spectrogram literacy remains an unusual edge case for multimodal architectures, but it exposes real differences in visual pattern recognition across frontier models.
- –Reasoning scaling inversion: Carbon Medium edging out Carbon High highlights diminishing or negative returns on extended test-time compute for dense visual classification tasks.
- –Spectrogram interpretation limits: Leading models still struggle to decode acoustic nuances solely through visual frequency maps, exposing blind spots in current vision encoders.
- –Pareto inefficiency: Neither Terra nor Hydrogen established a viable tradeoff between accuracy, latency, and cost on this benchmark.
- –Benchmark specialization: Adam Holter's Priors suite continues to surface niche evaluations that resist benchmark saturation and prevent artificial memorization.
DISCOVERED
1d ago
2026-11-15
PUBLISHED
1d ago
2026-11-15
RELEVANCE
AUTHOR
AdamHoltererer