SOB Benchmarks Value Accuracy, Not Just JSON

// 45d agoBENCHMARK RESULT

SOB Benchmarks Value Accuracy, Not Just JSON

Structured Output Benchmark (SOB) argues that schema-valid JSON is the wrong bar for structured generation, because models can still hallucinate values, misorder arrays, or map dates incorrectly. Its leaderboard adds value accuracy, faithfulness, path recall, structure coverage, and perfect-response scoring across text, image, and audio.

// ANALYSIS

The benchmark is directionally right: if downstream systems depend on exact fields, “valid JSON” is mostly a low-effort pass condition, not proof of useful extraction.

–Value accuracy is the metric that matters for real workflows like invoice parsing, UI automation, and document extraction
–The JSON-pass vs value-accuracy gap is the real headline, because it shows most models are optimized for shape, not correctness
–Multimodal coverage matters here: structured output failures are not just text problems anymore
–The open-source setup makes this more credible as a community reference point, not just a vendor marketing chart
–GLM 4.7 placing near the top suggests open models are closing in on frontier systems for deterministic tasks

// TAGS

structured-output-benchmark-sobbenchmarkstructured-outputmultimodalopen-sourceresearch

DISCOVERED

45d ago

2026-04-29

PUBLISHED

45d ago

2026-04-28

RELEVANCE

9/ 10

AUTHOR

404llm

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

OPEN SOURCE51m ago

Music Assistant is an open-source media library manager that unifies your music streaming services and smart speakers into a single self-hosted server.

Music Assistant functions as a centralized hub for your music library, connecting various streaming services and local media directories to a wide range of smart speakers. Designed to run continuously on low-power devices like a Raspberry Pi, NAS, or Intel NUC, the server aggregates searches, manages playlists, and orchestrates multi-room synchronized audio playback across disparate hardware systems.

NEWS54m ago

Developers compare Claude Fable 5 and ChatGPT 5.5

A social media inquiry by @droidbuilds asks the developer community to identify distinct capabilities that Anthropic's new "Mythos-class" Claude Fable 5 model can perform which OpenAI's ChatGPT 5.5 cannot. This query highlights the intense competition and performance comparisons between the latest frontier AI models as developers benchmark their long-horizon agentic workflows, complex coding tasks, and multi-step execution.

POLICY1h ago

Amazon reports trigger Fable 5, Mythos 5 ban

Following reports that security researchers successfully jailbroke Anthropic's restricted Mythos 5 AI model, the U.S. Department of Commerce issued an emergency export-control directive restricting access for foreign nationals. Because Anthropic was unable to dynamically filter users by nationality, it requested its cloud partner, Amazon Web Services (AWS), to disable global access to both Fable 5 and Mythos 5, sparking debate over government oversight of frontier AI models.