OPEN LINK ↗
// 78d agoBENCHMARK RESULT
LLMs flub surgeon-riddle prompt
A LocalLLaMA discussion claims Gemini 3.1 Pro was the only tested model to answer a surgeon riddle consistently, while several local models and GPT-5.4 reportedly defaulted to the classic memorized response. The more interesting takeaway is that commenters argue the riddle was phrased differently enough that the “obvious” answer may itself be wrong, making this a useful anecdote about pattern-matching versus careful reading.
// ANALYSIS
This is less a clean model win than a neat stress test for whether LLMs read the prompt in front of them or retrieve the famous answer they have seen a thousand times before.
- –The post hinges on a tweaked version of the classic surgeon riddle, which commenters say removes the original contradiction entirely
- –If a model jumps straight to “the surgeon is the mother,” it may be recalling benchmark-like training data instead of parsing the exact wording
- –Several replies note that stronger prompting or asking models to cite the relevant text can flip the result, which points to prompt sensitivity as much as raw capability
- –It is an anecdotal Reddit test, not a rigorous eval, but it highlights a real weakness formal benchmarks often smooth over: overfitting to familiar reasoning patterns
// TAGS
gemini-3-1-prollmreasoningbenchmarkprompt-engineering
DISCOVERED
78d ago
2026-03-10
PUBLISHED
81d ago
2026-03-07
RELEVANCE
6/ 10
AUTHOR
jslominski