Jamie Simon calls for physics-style science of deep learning

Deep learning is currently in its "pre-Newtonian" phase, where we have the engineering to build cathedrals but lack the materials science to explain why they stand.

• –Advocates for "dots-on-curves" theory—mathematical models like Scaling Laws that make quantitative predictions about training dynamics.
• –Criticizes current research for either being "Step A only" (speculative philosophy) or "Step B only" (mere engineering), calling for a balance of both.
• –Argues that over-mathematization often obscures insight, suggesting a "physics" approach of non-rigorous but illuminating arguments is more valuable for discovery.
• –Highlights the importance of "Hail Mary" hypotheses and Mechanistic Interpretability in discovering the internal logic of black-box models.