AI Has Never Seen a Single Letter. It Reads Tokens.

In progressCount Grey Mattervideo2026-10-29read planted ai llmshistory

Synopsis — AI-drafted from Dan's notes

The second of the persona channel’s “lantern lectures”, a 22-minute explainer built on the strawberry question: why a model that drafts contracts miscounts the Rs. The Count’s answer is that a language model never sees letters. A tokenizer cuts text into pieces, often whole words with their leading space, and each piece becomes a number; the model maps the number to a learned place among all the others (an embedding), works on those places layer by layer, and comes out with a probability for every token in its vocabulary. He shows a small open model giving four Rs as its first guess. Tokens need not be text: images go in as 16-by-16 patches.

The history runs from Markov’s 1913 count of 20,000 letters of Eugene Onegin (two symbols, vowel and consonant, and far fewer vowel pairs than chance predicts), through Shannon’s letter and word chains, to Philip Gage’s 1994 byte pair encoding, a compression trick that Sennrich, Haddow and Birch borrowed in 2015 for translation and GPT-2 ran on raw bytes. Vocabularies grew from two symbols to about 50,000 and then about 200,000 (GPT-4o’s; GPT-4’s was about 100,000).

Then the costs. Because the vocabulary was built from frequency, mostly English text, other languages pay more tokens for the same sentence; Petrov and colleagues at Oxford found gaps of up to 15 times, and newer tokenizers have narrowed but not closed them. What sits inside a token is hidden: Gwern suspected tokens for GPT-3’s bad rhymes, and Singh and Strouse showed that forcing digits into groups from the right with commas raised GPT-3.5’s addition accuracy from about 76% to 98%. Yet models rebuild what they are not given, gluing multi-token words back together in early layers and carrying a rough record of each token’s spelling, which is why spelling a word out first usually fixes the count. OpenAI announced in April 2026 that ChatGPT could finally count the Rs in strawberry, and cranberry still failed. He closes on Meta’s Byte Latent Transformer, which reads bytes in variable patches, as a sign tokens may not last.

Connections