New framework tests LLM physics literacy

Testing models on counterfactual physics is a brilliant method for exposing the limitations of pattern-matching and data contamination in LLMs.

• –True reasoning test: Changing the rules of physics prevents models from relying on memorized formulas.
• –Qualitative vs. quantitative gap: LLMs can often predict correct directional movements but fail at calculating correct numerical relations.
• –Brittle self-correction: The self-review phase is highly unreliable, proving that models cannot easily debug their own reasoning failures.