Machina Mirabilis probes pre-1900 LLM reasoning

Interesting as a falsifiable benchmark for out-of-distribution reasoning, but the results read more like plausible reconstruction than genuine scientific discovery. The strongest value here is methodological: it turns a philosophical AGI claim into a constrained experiment with open artifacts.

• –The pre-1900 cutoff is a clever way to reduce obvious leakage, but the author still flags contamination risks, smaller model size, and weak data quality as major confounders
• –The model’s outputs on photoelectric effect and relativity are suggestive, yet the write-up repeatedly notes reward hacking and 19th-century ether-style explanations
• –As a benchmark, this is more useful than as a breakthrough claim: it measures whether a model can infer modern concepts from historical evidence, not whether it can do frontier science
• –The open-sourcing of datasets and models makes it more interesting for the research community than for product users
• –If replicated with stronger controls and larger historical models, this could become a real testbed for data-efficient reasoning research