Is Power-Seeking AI an Existential Risk?

Carlsmith’s report sets out the case for AI as an existential risk as a chain of six premises and puts a probability on each. It went public in April 2021 and reached arXiv in June 2022.

The systems he worries about are a specific kind, which he calls APS. They have advanced capability (they outperform the best humans at tasks that confer real power, such as science, engineering, strategy, hacking and persuasion), agentic planning (they make and carry out plans in pursuit of objectives) and strategic awareness (their model of the world is good enough to see what gaining and keeping power would do). The argument is about those systems, not about AI in general.

The hinge is a hypothesis he calls instrumental convergence. If such a system is misaligned at all, and some of its misaligned behaviour involves strategic planning in pursuit of problematic objectives, then by default we should expect it to seek power in ways its designers didn’t intend, because power is useful for almost any objective. He states this as a default, not a law.

Then the chain, for the years up to 2070, with each step conditional on the ones before. It becomes possible and financially feasible to build APS systems: 65%. There are strong incentives to build them: 80%. It is much harder to build ones that stay aligned in practice than ones that don’t but are still attractive to deploy: 40%. Some deployed systems seek power in misaligned, high-impact ways, which he pegs at more than a trillion dollars of damage: 65%. That scales to the permanent disempowerment of roughly all of humanity: 40%. That counts as an existential catastrophe: 95%. Multiplied together the figures come to about 5%. A note dated May 2022 says his overall estimate had since risen above 10%; it gives no new figures for the premises.

He gives reasons to think safety is harder here than for ordinary technology. The systems are hard to understand from the inside. A strategically aware system has reasons to pass tests it would otherwise fail. And where a failed bridge does limited, contained harm, a failure here can spread and become harder to stop, the way an escaped virus does. He also gives the other side. He expects warning shots from weaker systems, says that seeking power is a long way from getting and keeping it, and thinks people might well correct failures before they become permanent.

He is open about the limits of the method. The product of a chain depends on how many links it has, so an appendix restates the argument in three premises (65%, 35%, 20%) and reaches the same figure. He calls the numbers unstable and subjective, says the whole picture feels like a very specific way things could go, and notes that the people he talks to most are selected for being worried about this.

Near the end, Carlsmith says the systems in question may be moral patients and that we have very little idea how to tell. His concern, he says, is disempowerment that nobody chose, not a human right to rule over what we build.