LLM Survey Bots Mimic Humans, Miss Nuance

This is a useful reality check on synthetic respondents: LLMs can imitate the shape of survey output, but that is not the same as recovering real human beliefs.

• –The study suggests models are good at generating conventional, internally consistent answers, which can fool you into thinking you have signal
• –The real value of human surveys here is the counterintuitive distribution of responses, and that is exactly what the synthetic sets flatten out
• –For researchers, synthetic respondents look more defensible as a pre-fieldwork probe or post-fieldwork sanity check than as a replacement for panels
• –The paper strengthens the case for explicit validation protocols and reporting standards before synthetic data is treated as evidence
• –If multiple models converge on similar answers, that may reflect shared training priors more than any true read on the population