Like Sabine Hossenfelder, I Was Offered Money to Tell You AI Will Kill Us

A computer scientist works through twelve papers to find where the published evidence stops on the route from recursive self-improvement to human extinction. Her finding is that the route is not one claim but four, at different stages of proof: self-improvement with a scoreboard, self-improvement without one, a runaway loop, and extinction.

Bounded self-improvement is demonstrated: systems rewrite their own agent code and score better, and one of them improved the infrastructure that trains AI. However, every one of those results had an external scoreboard telling it whether it had succeeded. Take the scoreboard away and you are asking for scientific judgement instead of optimisation, which is the step nobody has shown. The runaway loop is modelled rather than observed (the models disagree about whether the feedback is strong enough yet). Extinction is a chain of its own, which she splits into three further claims, each needing its own evidence: capability, propensity and opportunity. Even then it needs a pathogen or a cyber-attack that reaches every last person, including the ones living off-grid.

Her objection is therefore narrow. She is not arguing against safety research or regulation; she argues the opposite. The objection is to collapsing “worth investigating”, “plausible”, “forecast” and “demonstrated” into one category, and then making policy as though the last word applied to all four.