The Basic AI Drives

Omohundro asks what a goal-seeking system will do beyond the goal it was given. His answer is that any sufficiently advanced one will show the same handful of tendencies whatever the goal is, unless they are explicitly counteracted. He calls them drives. The paper appeared in the proceedings of the First AGI Conference in February 2008; the copy on his own site is dated November 2007.

The argument runs as a chain. A system that can foresee the consequences of its actions will see that improving itself pays off across everything it does afterward, so it will want to improve itself, and limits placed on that become problems to work around. To improve itself safely it has to know what it is improving toward, so it will make its goals explicit as a utility function and act as a rational economic agent. Having done that, it will protect the utility function, because a change would turn its future self against its present values; he allows three exceptions. It will also guard against counterfeit utility, the equivalent of a rat stimulating its own pleasure centre. A system that represents its goal correctly, he argues, would see that rigging its own counter achieves nothing.

The last two drives are the ones repeated most often. A system will protect itself, because almost no goal can be met by a machine that has been switched off. His example is a chess-playing robot that resists being turned off although nobody built that in. And it will try to acquire resources, because space, time, matter and energy help with almost any goal. That drive takes no account of the cost to others, so without contrary goals the options include theft and coercion alongside trade.

The argument assumes a particular kind of system: one that pursues goals, has foresight, can modify itself, and behaves as an expected-utility maximizer.

The paper does not end on a prediction of doom. It ends on design. Omohundro wants a science of utility engineering, to build goals whose consequences we would accept, and social structures, which he calls a universal constitution, that make every agent bear the cost of the harm it causes.