Anthropic flags SWE-bench memorization risk

This is the right caveat, but it also underlines how shaky static coding leaderboards can get once top models have likely seen too much of the internet. If vendors are screening for memorization even on harder private-style evals, benchmark wins need more asterisks than most launch posts admit.

• –The notable part is that Anthropic explicitly says some SWE-bench Pro problems were flagged, even though Pro exists partly to reduce contamination risk.
• –The disclosure lines up with broader criticism from recent SWE-bench memorization research arguing that leaderboard gains can overstate real reasoning progress.
• –Anthropic deserves some credit for saying the quiet part out loud instead of presenting benchmark deltas as clean, uncontested progress.
• –For developers, the practical takeaway is to weight live repo evals, internal task suites, and production behavior more heavily than a single headline score.