Dev benchmarks 40 models on personal prompts
Developer Avid shared an automated prompt workflow that evaluates 40 AI models across 13,192 personal prompts from 3,151 sessions. The prompt sorts recurring tasks into six domain buckets, runs blind model-versus-model evaluations, and generates a custom ROUTING.md file with fallback tiers to cut inference costs.
Public LLM benchmarks measure synthetic tests rather than real developer workloads, leading teams to overpay for frontier models on routine tasks. Auditing personal prompt histories to generate automated, blind routing tables provides an objective, cost-effective alternative to public leaderboards.
- –Grounded in real data: Auditing 13,192 real user prompts ensures evaluation metrics mirror actual day-to-day requirements instead of synthetic coding puzzles.
- –Blind cross-evaluation: Removing model identifiers and letting LLMs grade candidate outputs blind eliminates brand bias from the selection process.
- –Task-based specialization: Dividing tasks into buckets like design, coding, and marketing allows cheaper models to win routine workloads while reserving frontier models for complex reasoning.
- –Portable routing tables: Generating a standard ROUTING.md format integrates cleanly with multi-model routing proxies like Codex Router.
DISCOVERED
2h ago
2026-11-22
PUBLISHED
2h ago
2026-11-22
RELEVANCE
AUTHOR
Av1dlive