OpenRouter launches Ori Eval for coding agents

Automated, codebase-native evals are the missing piece in LLM application engineering, shifting developers away from generic public benchmarks toward task-specific empirical validation.

• –Automatically discovers model calls in code and writes targeted eval suites based on specified criteria.
• –Validates agent tool usage using strict assertion checks (e.g. tools invoked or avoided) alongside LLM-as-a-judge scoring.
• –Integrates into CI/CD workflows like GitHub Actions to catch prompt or model regressions before deployment.
• –Standardizes test environments by pinning model harness, effort settings, and execution parameters.