Rorschach tests expose LLM contamination limits

Hot take: this is much better at revealing how multimodal models reproduce culturally saturated response patterns than at revealing anything clean about visual cognition.

• –The strongest objection is contamination: the standard cards, manuals, and common interpretations are public and likely embedded in training data, so the task is partly retrieval and pattern completion, not fresh perception.
• –The setup is under-controlled for a serious inference: public web interfaces, default vendor behavior, and apparently single-pass administration make the outputs hard to interpret causally.
• –Even so, the paper has limited exploratory value: it shows that LLMs can generate coherent, human-like narratives around ambiguous images and can be scored by psychometric frameworks, which is itself a useful stress test.
• –The scientifically stronger version would use novel or synthetic ambiguous stimuli, multiple repeated trials, API-level decoding control, preregistered coding, and baselines that separate image understanding from memorized psychometric language.
• –What it mostly demonstrates is advanced statistical pattern matching plus learned psychometric priors, not evidence that the models have an “inner world” in any human sense.
• –Papers like this get through peer review when they are framed as pilot or hypothesis-generating work, but the interpretation often outruns the evidential strength of the design.