PiBench mocks frontier LLM rote recall
Ironwood AI founder Adam Holter shared PiBench, a satirical evaluation testing frontier models on reciting 3,000 digits of pi from memory without tools. The mock scorecard highlights the limits of rote recall and unhelpful alignment refusals across fictional Claude and GPT variants.
While framed as a meme about reciting 3,000 digits of pi, PiBench skewers the recurring obsession with evaluating large models on massive memorization rather than true reasoning.
- –Rote memorization of mathematical constants exposes synthetic benchmark contamination, where models simply regurgitate training corpora instead of computing algorithmic outputs.
- –GPT-6.1 Terra’s simulated refusal highlights how safety tuning and RLHF frequently cause over-alignment, making models refuse benign factual recall queries.
- –Mocking unreleased model tiers like Claude Fable and OpenAI Sol points to growing developer fatigue around confusing sub-versioning and naming proliferation from top labs.
- –Holter’s human baseline score of 26 emphasizes that sequence recall is an empty metric for real-world intelligence and agentic software development.
DISCOVERED
36m ago
2026-11-16
PUBLISHED
1h ago
2026-11-16
RELEVANCE
AUTHOR
AdamHoltererer