IDP Leaderboard pits 16 document VLMs

This is more useful than yet another one-number model ranking because it exposes raw predictions, failure modes, and cost/performance tradeoffs on real document workloads. For AI teams building OCR, KIE, or table pipelines, that kind of transparency matters more than a glossy benchmark win.

• –The standout product feature is the Results Explorer, which shows model outputs beside ground truth instead of hiding behind aggregate scores
• –Gemini 3.1 Pro leads overall, but cheaper variants like Flash and Sonnet stay surprisingly close on extraction-heavy tasks, suggesting reasoning is where premium models still justify their cost
• –GPT-5.4’s jump over GPT-4.1 is significant, especially on DocVQA and table extraction, making document understanding one of the clearer areas of recent model progress
• –Sparse unstructured tables and handwriting OCR remain stubbornly hard, which is exactly the kind of reality check production teams need before trusting vendor accuracy claims
• –The benchmark is open, reproducible, and linked to public datasets and code, which gives it more credibility than closed vendor bakeoffs