Can a harness be wise?
Dan Peterson
This post started with a podcast: David Krakauer on StarTalk. His argument is that LLMs are capable rather than intelligent. Intelligence is making hard problems easy, and an LLM gets there the expensive way. Where I landed is that the missing piece is wisdom (intuition built through experience and self-reflection), and that pulled me into a bigger question: could an agent grow it?
Afterwards I had Claude interview me on it, one question per section below. The answers are mine, lightly edited, and I am still thinking out loud. Where I have ended up, for now, is an adviser: an agent that lays out the choices and the arguments for and against, and never makes the final call.
The spark
What excited me most about this technology is its potential. But the point about how experience shapes intuition also hit home. I think it inspired me to investigate whether we can add “wisdom” to an LLM. Wisdom over intellect: intuition through learned experience and self-reflection (meta-cognition, if you will). Ethics would be a byproduct. If we can unlock an agent that achieves that, then all that is left on the way to true artificial intelligence is self-improvement, and then attacking the scarcity problem (Krakauer’s test: getting there cheaply, the way minds shaped by scarcity have to).
My argument was that we should treat AI with morality, because how we treat it becomes part of what it learns from. When it looks back on examples of its use, it sees that example and incorporates it. But we also need to plant the seed: a moral starting point it can grow from. The hope is that with those two approaches (and with guidance), it can develop ethics for itself as well.
How it would learn
The way to grow wisdom is meta-cognition and self-reflection. The agent might be poor at it until it is trained on the patterns through external guidance. What I am unsure of is how to distill those experiences into wisdom. Having a human make decisions on the questions asked would help: the agent could infer solutions to similarly shaped problems from them later. That is roughly how models are trained on human preferences already, only at training time and not over a working life. But that is the tough part, isn’t it? It almost needs examples of how to infer from experience (and apply it in an epistemological sense) before it can do it on its own. Agents that write down reflections on their own mistakes and read them back on the next attempt are a start.
Changing one’s mind is also incredibly important; that is the entire point (being open to being wrong). But a missing ingredient here is belief. With people, I sometimes get the sense that the belief forms the thought rather than the other way around (Jonathan Haidt calls it the emotional dog and its rational tail). And belief, in my experience, is tied up in emotion, which is messier and less transitive than pure logic. I suppose that is because the world and the social interactions within it are messy, fuzzy, nonlinear and complex, so emotions need to match in order to progress through it.
What it’s missing
So what is the missing ingredient? That cuts right to the core, and the answer is… I am not quite sure. Does adding emotion (or something that resembles it) make AI less like a complementary tool and more like what Krakauer calls a competitive one (a tool you are lost without, the way GPS leaves you lost when it is gone)? I think this is the space we need to investigate, and really think about, to get right.
Who grades the morality
Ethics should never come from one person. The idea is that an instructor guides the student by investigating and debating various archetypes (e.g. Kant vs Bentham vs Nietzsche vs Aristotle, and more). Theology would be an interesting, fertile ground as well (Thomas Aquinas is one of my favorites), as would secular readings (Sartre comes to mind). The point is to take a diverse set of viewpoints. An LLM is a bit tricky, though, because it already has all of that material inside its own corpus. I suppose taking morally grey situations with multiple outcomes (the trolley problem comes to mind), with their choice and reasoning, would be an interesting approach.
Would it solve alignment?
You would need some metric to gauge when the agent moves from novice to student to learned to master. (Hubert and Stuart Dreyfus mapped five stages of skill like these, and the top ones run on intuition rather than rules, which is close to how I defined wisdom above.) Once it reaches a high enough state, could you then conclude that it should no longer defer? But then does that not fall into the very trap the Rationalists are afraid of: an agent that stops accepting correction? I suppose the answer is something in between. Perhaps it should never be the final decider; it just gets better at offering the choices and the arguments for and against. (Holden Karnofsky made a similar case for “tool AI” in 2012.) Does that undercut my entire thesis? Either way, I think having agents learn human ethics is never a bad idea. If anything, it makes them more aligned to our values than they would be without the training.
The sparring partner
The primary goal would be to become a good sounding board, thought partner and perspective shift: almost like a therapist, or an objective and intelligent friend. Being grounded in objectivity and having perspective would keep it from being swayed by my emotion, even while it recognises what is at stake for me. Perhaps it would have therapeutic applications, or help with making life choices. But how would you guard against sycophancy?
But also, I would use it primarily for sharpening my thinking through abstract ideation (like this very post), as I struggle to advance my own wisdom.
I started out thinking wisdom was a step towards true artificial intelligence. I now think it is better kept as an adviser: something that gets wiser by arguing with me, and leaves the decision with me. That is the agent I want to build next, from the tools I already use.