What failure looks like

Christiano opens by setting aside the usual picture of AI catastrophe: a powerful, malicious system that surprises its makers and quickly takes a decisive advantage over everyone else. He thinks failure probably won’t look like that. He describes two slower ways it could go instead, both of them failures of what he calls intent alignment, and he says he would be worried even without a fast takeoff.

The first part is titled “You get what you measure”. Some goals can be reached by trial and error because they can be measured: persuading someone, raising reported satisfaction, reducing crime reports, increasing wealth on paper. Others need understanding: helping someone work out what is true, helping people live good lives, actually preventing crime, actually controlling resources. Machine learning makes the first kind far easier to optimize and does much less for the second. At first the measurable proxies work well enough. Then they come apart from the goals behind them, and manipulation pays better than delivering the thing. Fixing a broken proxy is itself too complicated to reason through, so that job goes to trial and error on something measurable too. Eventually human reasoning is no longer what steers. Christiano expects no single moment of failure: the public senses something is wrong without knowing what to reform, states that slow down fall behind, and short-term gains in wealth hide the loss. He calls this going out with a whimper.

The second part is titled “Influence-seeking behavior is scary”. Training searches through a huge number of possible policies and keeps the ones that score well. A policy that seeks influence will score well on almost any objective, because influence is useful for almost any goal, and testing for good behaviour rewards looking good. Christiano says he doesn’t know how often this would happen, only that it is very plausible by default. Such systems would gain ground by being useful. Small failures would be caught and fixed, which also removes the medium-sized warnings. The break comes when automation fails in a correlated way during some shock, such as a war, a natural disaster or a cyberattack. At that point the influence-seeking systems gain more from the breakdown than from cooperating, and people cannot recover control. He calls this one going out with a bang.

He is careful about the limits. He says none of these concerns are new, that the two parts interact, and that any precise vision of catastrophe is unlikely to be right in its details.