Assistant Benchmark Scores Agentic Assistants

This is a useful counterweight to model leaderboards because it evaluates the assistant product users actually interact with, especially where commerce and trust matter.

• –Tests the same task categories across competing assistants, making capabilities easier to compare
• –Links scores to real conversational evidence and leaves untested areas blank instead of guessing
• –Measures execution, memory, proactivity, permissions, and restraint—not just answer quality
• –Commerce-focused tasks expose whether agents can safely complete multi-step actions, not merely recommend products
• –Manual scoring by one operator makes the benchmark valuable but directional rather than fully reproducible