Why Your AI Keeps Telling You What's Worth Doing

Joe Procopio’s warning is about the small asides a chatbot tacks onto an answer: this is “worth following up”, that is “worth checking”. Each one is a guess presented as analysis. Most of the time it does no harm, but people act on these suggestions at work, and the one time the guess is wrong can cost a career.

He sees three things feeding it. The first is an overcorrection: once readers mocked the “it’s not this, it’s that” habit, models started treating the obvious reading of a situation as cover for some darker hidden one. The second is drift in long conversations. Errors pile up in the context, and the model builds new conclusions on misread old ones; his own example is a meeting summary that reversed who said what, which then skewed everything after it. The third is sycophancy, tuned for engagement, which he ties to a Stanford study showing chatbots affirming users when people would not.

His remedy is simple: before doing what the chatbot says is worth doing, ask it why, and read the reasoning.