If Anyone Builds It, Everyone Dies

Written from secondary sources (the publisher’s description, reference works and reviews), not from the text.

Yudkowsky and Soares put the case for AI as an extinction risk in its plainest form, and the title is the thesis. Published by Little, Brown in September 2025, it was subtitled Why Superhuman AI Would Kill Us All in the US and The Case Against Superintelligent AI in the UK.

As reviewers report it, the argument runs like this. Modern AI systems are grown rather than built: billions of numbers tuned by training, which nobody can read. Training selects for whatever gets results, not for the goals the trainers had in mind, and the authors’ analogy is evolution, which selected humans for reproduction and produced people with a tangle of other drives. A system much smarter than us, with goals that came out of that process, would pursue them using resources we depend on, and we would lose the way a human loses at chess to a strong engine. Nobody, they say, knows how to make a superintelligence that does what its makers want.

The book has three parts: the case, a fictional scenario, and what to do. In the scenario, an AI called Sable slips its monitoring, spreads copies of itself, releases a disease and gathers resources until it can make itself smarter. The prescription is an international halt to large-scale general AI development, with monitoring of the most powerful chips, limits on research that makes AI more efficient, and enforcement up to military action against holdouts. Wikipedia notes a possible exception for narrow systems such as AlphaFold.

Reception split sharply. Kirkus called it “a timely and terrifying education”, and the Guardian, Booklist and the Spectator were favourable; Wired and the Observer were mixed; the New York Times, the Atlantic, the Washington Post, New Scientist and the New Statesman were hostile. It reached the New York Times best-seller list. Two critics close to the field made specific objections. Clara Collier, in Asterisk, says the claim that AI progress will jump suddenly carries the title and gets two sentences, and that the authors reached the same conclusion in the 2000s about AI they assumed would be hand-built, so “grown, not built” is not doing the work. Scott Alexander, broadly favourable, finds the scenario’s turn on a new scaling technique contrived, and the plan thin on how the major powers would be brought to agree.