182-Paper Review Finds Synthetic Participants Fail

Hot take: the headline is directionally right, but the more useful claim is narrower: synthetic participants are a tooling shortcut, not a research substitute.

• –If your question is “what might people say?”, LLMs can be a fast first pass.
• –If your question is “what do real users actually do?”, they are the wrong instrument.
• –The failure mode is not just bad prompting; it is structural mismatch between prediction-driven text generation and human behavior.
• –This should change how teams use them: ideation and test design are fair game, decision-making and validation are not.
• –The real risk is overconfidence: synthetic users can produce polished outputs that feel representative while masking missing populations, weak sampling, and false certainty.