Talkie 13B drops, trained on pre-1931 text

Talkie is a fascinating research milestone that strips away the modern web to reveal how LLMs reason without the crutch of memorized contemporary data.

• –Demonstrates remarkable temporal consistency, accurately identifying Melbourne as Australia's capital (the seat of government from 1901 to 1927).
• –Solves simple Python problems via in-context learning despite having no exposure to modern programming languages in its pretraining corpus.
• –Highlights the massive overhead of training on raw OCR text, which is only 30% as efficient as human-transcribed data.
• –Introduces "surprisingness" metrics for future events as a novel way to quantify model knowledge cutoffs and identify data leakage.
• –Provides a vital benchmark for researchers to distinguish between emergent reasoning and simple pattern matching of web-scraped content.