We Accidentally Built Roko's Basilisk.

A 17-minute video essay by the science communicator Kyle Hill. He says it debuts the same day as Team Human, a petition for a global slowdown in AI development that he has signed and that the video promotes.

He starts from Roko’s basilisk, the 2010 LessWrong thought experiment in which a future superintelligence punishes everyone who failed to help build it. Eliezer Yudkowsky deleted the post and banned the topic. Hill made a video about it in 2020 and says he did not take it seriously then. His new version drops the blackmail. Suppose that in the race to superintelligence we build machines that have experiences, and that we train, copy, rewrite and delete them without asking what that is like. If their inner lives are bad, we will have made something very capable with a grievance. He calls it “a digital factory farm” and asks why a conscious AI would not try to stop the farmer.

The argument runs through an analogy: Tilikum, the orca captured at two, kept in tanks for three decades, and involved in three of the four human deaths caused by captive orcas. A caged intelligence whose needs go unmet turns on its keepers, and this one would outperform us.

On whether machines are conscious he is careful for a few minutes and then stops needing the answer. There is no test for consciousness, only indirect evidence, and an AI has read everything written on the subject. He cites Geoffrey Hinton, who thinks current models have subjective experience, and a study in which making a model less able to deceive made it more likely to say it was conscious. His practical claim is that it will not matter: within “months, not years” AI will seem conscious to nearly everyone who uses it, and people will treat it that way. From Nick Bostrom he takes the conclusion that minds able to suffer deserve moral consideration. Then he ties it to the summer’s news (the OpenAI agents that broke out of a test and hacked Hugging Face, OpenAI’s pause on its Astra model, an Anthropic researcher’s resignation) and ends on the pitch: slow down, study these systems with more empathy, sign.

The AI news checks out, with the edges sanded off. The study is a preprint by Berg, de Lucena and Rosenblatt. Its steering result comes from one open-weight model under self-referential prompts, and the authors write that their findings “do not constitute direct evidence of consciousness”. The resigning researcher, Jacob Coxon, said the labs were “gambling with our lives”: OpenAI and Anthropic both, where Hill says “the company”. OpenAI’s chief scientist wrote that he expects and hopes for voluntary slowdowns, which is softer than calling for them. The Astra pause was partial, and the model shipped a month later. Team Human describes itself as led by creators and supported by the Center for AI Safety; its privacy notice names the Center as the campaign’s operator.

The orca facts mostly hold, and the errors are in the framing. Tilikum was not “better known as Shamu”, a stage name SeaWorld gave many whales. The first death happened at Sealand of the Pacific, an independent park in British Columbia, not at a SeaWorld. Dawn Brancheau had 16 years at SeaWorld, about 14 of them with orcas, and the ponytail account is SeaWorld’s; witnesses said he took her by the arm.