ParetoBandit Keeps LLM Routing on Budget
ParetoBandit is a research paper and open-source adaptive routing system for multi-model LLM serving. It uses cost-aware contextual bandits to enforce dollar-denominated budget ceilings in real time, adapt to non-stationary changes in model pricing or quality, and onboard new models at runtime without retraining. The reported results show tight budget control, fast recovery from silent regressions, and low routing overhead, which makes it most relevant for production inference stacks that need dynamic cost-quality tradeoffs.
Strong systems paper with a practical angle: it is not just another routing benchmark, it targets the ugly production case where prices change, models regress, and new models appear midstream.
- –The budget pacing piece is the main differentiator; most routing work optimizes cost indirectly, while this enforces a real spend ceiling.
- –The non-stationary handling is credible and operationally useful if the reported adaptation behavior holds outside the authors’ setup.
- –The hot-swap model registry is a real deployment feature, not just an algorithmic flourish.
- –Best fit is infrastructure teams running multi-model LLM serving, especially where cost predictability matters more than squeezing the last point of quality.
- –Caution: this is still a paper-level result, so generalization to broader traffic mixes and vendor ecosystems remains the key risk.
DISCOVERED
50d ago
2026-04-07
PUBLISHED
50d ago
2026-04-07
RELEVANCE
AUTHOR
PatienceHistorical70