Local 4B LLM forecaster beats GPT-4 on cheap GPU

The real breakthrough here isn't the model itself, but the clever calibration pipeline that fixes the inherent overconfidence of small LLMs. It proves that aggressive post-processing on a 4B model can yield superforecaster-level accuracy without touching the cloud.

• –Raw LLM output scored a poor 0.25 Brier score, but applying a shrinkage formula toward a measured base rate improved it to 0.186
• –Runs entirely locally via llama.cpp taking just 3.4 GB VRAM, making it accessible on a $300 secondhand GPU
• –Includes a built-in RAG pipeline with stealth browser fetching for grounding predictions in current events
• –Shows particular strength in discrete events and market questions, though struggles with continuous macro time series data