Why We Haven't Reached Superintelligence, with David Krakauer
Dan PetersonSynopsis — AI-drafted from Dan's notes
A 78-minute StarTalk special edition, uploaded on 10 October 2026, with Neil deGrasse Tyson, Chuck Nice and Gary O’Reilly talking to David Krakauer, the evolutionary biologist who runs the Santa Fe Institute. Krakauer starts from a definition and holds onto it all the way through: intelligence is making hard problems easy. Every living thing has some of it. Something could be intelligent without being alive, and he spends the episode testing whether LLMs are that something.
His answer is no, or not yet. By his definition they are capable rather than intelligent. An LLM starts from language and skips everything underneath it: no body to keep alive, no unconscious, no curriculum. It learns everything at once, from everyone. His example is OpenAI’s September claim that a swarm of agents showed the forced 3D Navier–Stokes equations can blow up. He grants the result and objects to the method: thousands of agents and millions of dollars (he says tens of millions; the only figure reported is “millions”) to get there through text alone. That is eating soup with a fork, or fetching a glass of water by running twenty laps of the room first. Getting the water is capability. Getting it cheaply is intelligence. So he rejects “superintelligence” as a description of the current path, and guesses that alien minds, shaped by scarcity like ours, would look more like us than like LLMs. He adds that because models learn heuristics rather than algorithms, they will eventually get long arithmetic wrong, and people who have outsourced the skill won’t notice.
The idea from his own work is the difference between complementary and competitive cognitive artefacts, first set out in a 2016 essay. An abacus is complementary: trained users end up doing the sums in their heads without it. GPS is competitive: take it away and you are lost. That separates capability, which either tool gives you, from competence, which only shows when the tool is gone and which outsourcing leaves you dependent on whoever supplies the tool. Tyson pushes back with his slide rule, which he never missed once calculators came along. Krakauer’s reply is scurvy and power cuts, plus the intuitions about logarithms and significant figures that the old tool built and that may carry over into everything else.
The part that names the episode for this garden is the entropy argument. Coal and oil sat in the ground for hundreds of millions of years until the steam engine let us burn them. Books have been piling up in libraries since Sumer, and the transformer is the steam engine for that deposit: it mines the written record and turns it into conceptual power. Anthropic and OpenAI are the mining companies of the age, as BP and Shell were. Both revolutions produce entropy as their waste: heat from the data centres and, as the hosts put it, slop in the culture. The difference is that coal runs out and knowledge doesn’t, but knowledge can be fouled, because the mined output goes back into the deposit it came from. The library’s own mistakes are a record of how we got here; a model serves them up without that history. He ends on the case for AI as a tool, such as protein folding, where no tidy macroscopic theory exists and a network’s better scaling wins. What’s left for people, he says, is asking good questions.
Where I land
I think that this was an eloquent analogy - basically the corpus that LLMs were trained on encapsulates the aggregate of all human knowledge.
I do think it’s an interesting argument - but what if you conflate the architecture of American big-tech AI companies with those of more efficient open-source models (like MoE) or perhaps SLM (like Phi-4). Could they be considered intelligent if they are embedded in more localized, targeted applications. But I do think that one thing that LLMs lack that humans have, which is a critical requirement for intelligence (in how he defines it) is wisdom through experience. LLMs are currently missing that. They don’t learn, they are snapshots on a body of work ran through a series of prompts in a harness towards a goal. But by themselves, they don’t learn or improve. In a sense, you may make an argument about post-training being somewhat analogous to learned experience but eventually all the weights are frozen and the model doesn’t evolve past that point.
That would be a first requirement yes - if the models evolved and weights adjusted to learned experience that would make it closer. But I also accept his point for efficiency. It’s an interesting POV. I suppose all life (and thus all biological intelligence) has limits. Would artificial mind require some sort of limit?
Connections
Links
- https://www.youtube.com/watch?v=bcpQkvBCACw
- https://nautil.us/will-ai-harm-us-better-to-ask-how-well-reckon-with-our-hybrid-nature-236098
- https://nautil.us/the-kekul-problem-236574/
- https://andrew.ooo/answers/openai-navier-stokes-proof-10000-agents-september-2026-explained/
- https://arxiv.org/abs/2305.18654