Doctorow warns users hallucinate AI intent

Attributing intentionality to LLMs is the ultimate cognitive vulnerability in tech, blinding teams to the fundamental banality and unreliability of statistical text extrusion.

• –The sunset fallacy: Just as atheism reveals a sunset as striking physics devoid of conscious purpose, AI atheism strips conversational models of phantom intent, exposing them as pure mathematical correlation.
• –The illusion of expertise: AI outputs appear profound only to novices because models excel at smoothing over rote, repetitive basics while completely flattening the nuanced texture and edge-case mastery that real experts possess.
• –The danger of pseudo-personhood: Granting chatbots moral or cognitive status mirrors the perils of corporate personhood, creating legal and moral shields that justify resource extraction and environmental waste for soulless software.
• –The real hallucination problem: Software defects and hallucinations in LLM output are trivial compared to the human hallucination of believing an interlocutor exists on the other side of the prompt.