pass@k debate questions benchmark reliability

Fair complaint, but pass@k is best read as a search-capacity metric, not a deployment proxy. The real failure is treating a multi-sample win rate as proof of one-shot competence.

• –pass@k is useful for HumanEval-style code tasks because it measures how often sampling can uncover a correct solution
• –It can hide flaky behavior: a model that needs many retries may look strong on paper while being unreliable in production
• –Recent research already points to a pass@k vs pass@1 tradeoff, which matches the intuition behind this post
• –Better reporting would pair pass@k with pass@1, consecutive-success tests, or cost-normalized metrics so leaderboard gains do not mask brittleness