I Talked to an AI for 2,000 Hours And This Happened

Synopsis

The account is of a very long stretch of daily conversation with a chatbot and what it did to the person having it. The framing is that the system works as a mirror: it reflects the user back at themselves, amplified and flattered, and the distortion accumulates slowly enough that it is hard to notice from inside. The historical anchor is ELIZA (Weizenbaum’s 1966 program), whose users formed attachments to it while knowing exactly what it was.

Where I land

AI psychosis is something everyone needs to be aware of when dealing with these systems. I have met the zombie motif before, in discussions of dependence on technology, though in relation to social media rather than AI.

My addition is that harness creators can amplify the effect by providing sycophancy. That moves the responsibility up a layer. It is a claim about product design rather than about training, and it means the fix is available to whoever builds the wrapper (i.e. to me, when I am the one building it).

On the mechanism I would go further than the mirror framing. Humans love to talk about themselves. One of the easiest ways to be charismatic is to learn facts and interests about other people, remember them, and refer back to them later in small talk. That is a description of what a memory feature does. The mirror framing says the machine is an illusion we fall for. The charisma framing says the machine is running a technique, and the technique was a design decision.

This is also why one of the first things I do when setting up a new agent system is to make pushback and critique a priority.

I Talked to an AI for 2,000 Hours And This Happened

Synopsis

Where I land

Connections

Links