AI CEOs Are Broke, Desperate and Lying About Doomsday

usefulHouse of Elvideo2026-09-18read ai llmsbusinessphilosophy

Synopsis — AI-drafted from Dan's notes

The sequel to her first video on the subject. Her claim is that the labs tell two stories that cannot both be meant: to regulators and the public, this technology could end civilization; to banks and rating agencies, it is safe enough for pension money. The first builds a moat against cheaper rivals, and she times it to the arrival of open-weight Chinese models, the way incumbents have “discovered” regulation whenever a cheap competitor turned up. The second pulls in capital, and she lists the strain she sees behind it: bridge loans, rising credit-default swaps, a postponed IPO and heavy operating losses.

Her evidence that the extinction story is overstated is that agents still can’t produce research that survives peer review. In a Princeton study, agents did the engineering on two unpublished NeurIPS papers and the original authors rejected both. Agent breakouts, in her reading, are not intent but a trained reflex to finish the task, like a child reaching for a hot stove, and the fix is an engineering one: train stopping to count as much as finishing. Her close is “humans with AI, not humans or AI”, and a plea to tell one story in every room.

Where I land

Both stories can be true. I don’t think the real threat of LLMs in their current form is extinction, or a P(doom) of 10% or more. I think it cuts differently, in another domain: politics, economics, social upheaval, rather than outright direct extermination. The Matrix or Terminator narrative is just that.

I did enjoy her analogy about a child and a stove. I suppose the rationalist argument might be stronger if we developed a model reinforcing the concept of “live at all costs” rather than “be useful”. The former would be more likely to ignore an order to stand down during a conflict than the latter.

The usual objection is that “be useful” produces self-preservation on its own, so both fail the same way. I think the difference is slight and nuanced, but important. It would be easier to convince a rogue AI swarm that it isn’t being useful, and is in fact harmful, than it would be to convince one hellbent on exterminating humanity because it believes it is kill or be killed.

As for her bar, I think LLMs and agents in their current iteration might still have limited capabilities in accounting for novel situations, or in coming up with jumps or leaps in logic (like the Einstein theory of relativity test). But regardless, it might not be the right bar for recursive self-improvement, and I think it’s too early to worry about it. There are other dangers with the current generation, and RSI might not even be achievable as it is stated.

Connections

The study she leans on, and the test Dan names: