Thematic Generalization Benchmark V2 raises latent-rule reasoning bar

This is a smart benchmark design because it targets the exact failure mode where models look impressive on broad similarity but miss the precise rule.

• –Anti-examples force models to separate “adjacent but wrong” patterns from the true latent mechanism, which is closer to real-world reasoning tasks.
• –V2’s stricter ambiguity/exclusivity filtering should improve signal quality versus noisy eval sets where multiple answers can be argued as correct.
• –The hard-subset framing helps reveal model-family blind spots, not just headline top-1 scores.
• –Benchmark contamination remains a long-term risk for public evals, and the author has discussed holding back some items to reduce overfitting pressure.