MothBench benchmarks local LLMs on consumer GPUs
MothBench is an open-source benchmark suite for local LLMs that tests `/v1/chat/completions`-compatible endpoints across logic, math, code, reasoning, instruction following, creativity, and long-context behavior. It supports Windows EXE and Python/CLI usage, tracks latency and TTFT, and produces scorecards with keyword-based and LLM-as-judge scoring. The project is explicitly aimed at local AI on consumer and prosumer hardware, with the launch post highlighting Radeon VII ROCm results using Gemma 4.
Hot take: this is more useful than yet another cloud-only benchmark because it measures the stuff local users actually feel: latency, TTFT, reproducibility, and judge-based quality on real consumer hardware.
- –The benchmark is broad enough to be practical, with 120 tests across 8 categories and multiple run modes for quick checks or deeper comparisons.
- –The focus on ROCm and non-CUDA hardware is the differentiator; that makes it relevant for AMD GPU owners who are usually under-served by mainstream evals.
- –The reporting looks solid for self-hosted experiments: HTML/JSON export, category breakdowns, radar charts, and run history make comparisons easier.
- –The LLM-as-judge layer is useful, but it also means results are partly model-dependent, so absolute scores should be treated as directional rather than final truth.
DISCOVERED
5d ago
2026-04-06
PUBLISHED
5d ago
2026-04-06
RELEVANCE
AUTHOR
GreenM0th