OPEN LINK ↗
// 45d agoOPENSOURCE RELEASE
Local 4B LLM forecaster beats GPT-4 on cheap GPU
PredictBot is an open-source forecasting tool that runs Qwen 3.5 4B locally on consumer hardware to predict yes/no event outcomes. By applying a custom calibration pipeline to shrink overconfident raw outputs, it achieves a 0.186 Brier score, outperforming GPT-4 on structured prediction market questions.
// ANALYSIS
The real breakthrough here isn't the model itself, but the clever calibration pipeline that fixes the inherent overconfidence of small LLMs. It proves that aggressive post-processing on a 4B model can yield superforecaster-level accuracy without touching the cloud.
- –Raw LLM output scored a poor 0.25 Brier score, but applying a shrinkage formula toward a measured base rate improved it to 0.186
- –Runs entirely locally via llama.cpp taking just 3.4 GB VRAM, making it accessible on a $300 secondhand GPU
- –Includes a built-in RAG pipeline with stealth browser fetching for grounding predictions in current events
- –Shows particular strength in discrete events and market questions, though struggles with continuous macro time series data
// TAGS
predictbotllmagentragopen-sourceinference
DISCOVERED
45d ago
2026-04-18
PUBLISHED
45d ago
2026-04-18
RELEVANCE
8/ 10
AUTHOR
Salok1