Google, Artificial Analysis Launch MKA Benchmark
Google DeepMind and Artificial Analysis launched MKA, a 200-task benchmark testing whether agents can turn visually dense files into professional reports, spreadsheets, presentations, and marked-up drawings across 10 domains. Models run in the standardized Stirrup harness, while scored tasks remain private and 10 public examples are available. Google announcement: https://blog.google/innovation-and-ai/models-and-research/gemini-models/multimodal-knowledge-agent-benchmark/
MKA is a meaningful upgrade from screenshot-based QA: it evaluates the complete work product, not merely an answer. Its results should be treated as directional rather than definitive, since agent scaffolding, turn budgets, and automated grading materially influence outcomes.
- –Covers 200 tasks spanning 10 industries and 83 visually intensive occupations.
- –Grades factual accuracy and visual design across reports, spreadsheets, slides, images, and text files.
- –Standardized tools, sandbox access, and a 250-turn limit make model comparisons more consistent.
- –Results expose major efficiency gaps, with tested agents averaging roughly 30–130 turns per task.
- –Developers should optimize file rendering, formula validation, visual inspection, and tool use—not just prompting. Artificial Analysis methodology: https://artificialanalysis.ai/evaluations/mka
DISCOVERED
16m ago
2026-11-11
PUBLISHED
47m ago
2026-11-11
RELEVANCE
AUTHOR
Theo - t3․gg
