Stopping The AI Safety Cult
Synopsis
The episode draws a line from 1990s transhumanist mailing lists to the safety departments of the current model labs. The people who now staff those departments came out of a single small community (rationalism, later merged with the longtermist wing of effective altruism), and the money that funded the community also funded the labs. Therefore the doom argument is not an independent conclusion that the industry arrived at. It is the founding ideology of the industry, arriving back as a warning.
The more useful half of the episode is the part that drops the genealogy. Zitron and his guests argue that “slowing down AI” is a useless demand because it bundles a harmless coding assistant together with the thing actually worth worrying about: an agent handed powerful tooling and left running unsupervised for days to see what it finds. Their case is that stopping those specific experiments would cost the labs close to nothing in revenue, which removes the usual objection. As such, the fight worth having is over vocabulary first (i.e. naming the practice precisely enough that a rule could apply to it) and over the practice second.
Where I land
I came into this thinking the safety statements coming out of Anthropic were hype. I do not think that any more. I now believe the people saying these things are serious and genuine, and I was wrong about their motives.
I still do not accept the conclusion. I do not believe LLMs are going to suddenly wake up “God”, and I do not think the rationalist movement or the effective altruist movement is the right way to approach questions this complex. However, rejecting the framework is a different act from rejecting the person, and I had been doing both. So I am going to read Less Wrong properly. What struck me is that Yudkowsky is essentially doing what I am doing here (i.e. thinking out loud about deep problems in public), and I owe it to my own position to understand his before I keep arguing against it.
One caveat, and it cuts against me as much as anyone. I do not think it is possible to root out your own biases no matter how hard you try to rationalize or reason them out. Confirmation bias and cognitive dissonance are not things you can turn off. Even my own approach of having an LLM pick apart my reasoning and supply counter-examples is not enough (although it is a fun exercise). How can you use your own mind to judge your own mind? The act of judging the validity of your own thoughts changes those thoughts while you are having them.
Finally, a terminology complaint that this episode sharpened for me. We need to stop lumping in AI with LLMs. They are useful. They are not the method for unlocking the answers to everything in the universe.
Connections
Related: The Biggest AI Fraud Is the One Nobody Is Investigating, The OWASP Top 10 for AI Agents (ASI Top 10), Countering Misuse of AI — Threat Intelligence Report, September 2026