EleutherAI re-releases Pythia-14M, 31M for interpretability research

Tiny models are the new frontier for mechanistic interpretability, offering a rare opportunity to fully map neural features before LLMs scale beyond our reach.

• –The update fixes a critical bug where previous iterations were mistakenly trained on deduplicated data, which had broken experimental consistency for researchers.
• –The PolyPythias expansion provides a massive dataset for studying training stability, which is often overlooked in favor of raw performance.
• –These specific sizes (14M and 31M) are the primary testbeds for Sparse Autoencoders (SAEs), a key technique for making black-box models transparent.
• –Recent research suggests these tiny models align better with human cognitive "surprisal" than their massive counterparts, making them essential for AI safety and alignment research.