System One Models Challenge LLM Legal Certainty

The paper’s strongest contribution is reframing legal AI from “pick an answer” to “know when not to pick one.” Calibrated abstention looks like a practical systems primitive, but the benchmark is not evidence of courtroom readiness.

• –Generative models selected an incorrect holding in 37.8–85.6% of missing-authority cases; the most cautious decision model reduced that to 30.1%.
• –Decision models expose a tunable tradeoff between wrong citations and unnecessary human review, unlike ordinary LLMs’ single answer output.
• –An agreement gate that accepts an LLM citation only when a decision model agrees can substantially reduce citation errors.
• –Distilling decision probabilities into Qwen3-8B produced a calibrated local model without labeled examples, pointing toward private deployment.
• –The test uses one English U.S. case-law benchmark and a constructed absence condition, so real-world retrieval, jurisdiction, and precedent changes remain open problems.