Watching a language model think

One word at a time

A language model does one thing, over and over. It reads the text so far and puts a probability on every token in its vocabulary for what comes next. A token is a word or a piece of one; Qwen3-1.7B knows 151,643 of them. Then one token is picked, added to the text, and the whole model runs again. That is all “writing” is.

It helps to feel this before looking inside. Try the figure: pick the word you think comes next, then see the model’s real odds.

Two things stand out. The model’s favourite is often a small, safe token. After “The Eiffel Tower is located in” it prefers “the” (37%) over “Paris” (25%), because “in the heart of Paris” is also a good sentence. And for open prompts its confidence is thin: “The cat sat on the” gives “mat” only 11.6%. Everything later in this note is about where those numbers come from and what happens when you sample from them many times.

TakeawayThe text you see is one draw from a spread-out bet. To understand a model you have to look at the bet, not just the draw.

Thinking in space: a map of meaning, and what the map gets wrong

Inside the model, every token starts as a list of 2,048 numbers called its embedding. Tokens with similar meanings get similar lists. Nobody can picture 2,048 dimensions, so we squashed them into three with a standard method (UMAP) and built a map you can fly through, in VR and in films. It is genuinely striking in a headset. Whole languages sit side by side: “Paris” floats beside 巴黎 (Paris in Chinese), “Rome” beside 罗马 and “Roman”. Code tokens form their own continent.

But a map is a projection, so we measured what it keeps. For every one of the 151,643 tokens we took its 15 truly nearest neighbours in the full 2,048 dimensions and asked how many are also among its 15 nearest points on the 3-D map.

The answer is 43% on average: about 6 of 15. Most of the rest are still nearby (72% are within the map’s nearest 100), so neighbourhoods survive. Distances don’t. For random pairs of tokens, distance on the map and true similarity have a rank correlation of 0.07, close to none. Two tokens close together on the map are probably related; two tokens far apart tell you almost nothing. “Paris” is a good example: the map kept “paris” and 巴黎 beside it but put France, London and Madrid somewhere else, and surrounded it instead with Bordeaux, Oslo and Antarctica. Latin-alphabet words and word pieces keep their neighbours best (47–51%); less common scripts do worst (27%).

Flying through the map in VR added something real: you notice the multilingual clusters and the code continent in seconds, which no table shows you. It also adds a second projection, because your screen or headset flattens the 3-D map again. So we treat the map as a way to find neighbourhoods worth checking, and do the checking in the full space.

Maps can also show structure forming. In The Click, a tiny transformer we trained to add numbers on a clock, (a + b) mod 113, first memorises its examples while its 113 number embeddings drift in a tangle. Thousands of steps later they snap onto four clean rings and its accuracy on sums it has never seen jumps from about 11% to 100%. That picture is a projection too, but onto directions the trained network itself uses, which is why it comes out so clean.

TakeawayUse a 3-D map to find neighbourhoods, never to read distances, and measure how much of the real structure it kept.

A prediction locking in, layer by layer

Our longest piece follows one prompt through Qwen3-1.7B: “The Eiffel Tower is in Paris. The Colosseum is in”. The model answers “Rome” with 97.4% confidence, then keeps writing on its own: “Rome. The Great Wall of China is in Beijing.”

Between input and answer sit 28 layers. Each layer reads a running state for every word, called the residual stream, and adds its notes to it. A technique called the logit lens lets you stop that stream at any layer and ask, “if you had to answer now, what would you say?” You run the half-finished state through the model’s own final output step.

For the first ten layers the lens reads noise. Then it says “city” (layers 10 and 11), then “Germany” (69% at layer 19). At layer 21 it flips to Rome (85.5%) and never lets go. The timing differs per word: “China” is settled by layer 12, because “Great Wall of” makes it easy, while “Beijing” waits until layer 26. In the same layer where Rome appears, one attention head (layer 21, head 4) puts 66% of its attention on “se”, a piece of “Colosseum”. That is a correlation we saw, not a cause we proved.

The 27B model, a hybrid with 16 ordinary attention layers and 48 memory layers, decides later and more sharply. Rome first leads at layer 57 of 64, about 89% of the way down, compared with 75% for the 1.7B. Just before that, its lens reads 哪个, Chinese for “which?”, at layers 52 to 56, as if it were posing the question. On a two-step question (“the engineer the Eiffel Tower is named after was born in the French city of…”) the 27B answers Dijon, correctly, while the 1.7B takes the shortcut to Paris (62%). The 27B’s hidden middle step, “Gustave”, shows up at layer 61, very late.

Our River of Thought loop shows the same moment from a distance: 4,624 tokens from sixty short passages flow through all 29 stages (the embeddings plus 28 layers) as silk threads, and the Colosseum prompt’s last word catches fire at layer 21. To be precise, those are sixty separate short texts drawn together, not one long document, and each layer is flattened onto the plane where the sixty passages separate best. The shape of the river is a projection; the layer where it ignites is a measurement.

TakeawayIn both models the decision is late and sudden, in the last quarter of the network. Early layers look like noise through this lens, which is a limit of the lens rather than proof the model knows nothing yet.

What attention shows, and where it misleads

Attention is how each word looks back at earlier words. It is the most visualised part of a language model, because it draws beautifully: arcs between words, with thickness showing weight. Some of what those arcs show is real and checkable. In Qwen3-1.7B, layer 4, head 10 is an induction head, a pattern copier: from the final “in”, it puts 89% of its attention on “Paris”, because “is in” was followed by “Paris” earlier in the prompt. On a standard repeated-random-token test it scores 0.91.

The biggest thing in any attention picture, though, is an artefact of how these models work. In layer 3 the model writes one enormous value into the first word, “The”, multiplying its size about 80 times. From then on, heads with nothing useful to do park their attention there. Averaged over every head and layer, 68% of all attention lands on that one word. It is called an attention sink, and it dominates every raw attention map. So we asked the same question three ways.

In all eight prompts, raw attention’s favourite word was the first word, at 52 to 67%, even with a hidden start token placed there to absorb it. Weighting attention by how much each word’s content can actually change the output shrinks the sink to 9 to 17%, and sometimes finds the right word (“dream” in the Paris prompt, “until” in the glass of water). The only causal test, though, is the knockout: run the model again with a word made invisible. Hide “out” in “The baker took the hot loaves of bread out of the” and “oven” falls from 99% to 0.2%, although raw attention gave “out” only 5%. Across all words, both attention measures rank importance only loosely like the knockout does (rank correlation about 0.42 each).

This changed how we built things. When we made the Guess the Next Word game, raw attention pointed at the first word even with a start token added to soak it up, so the game uses the weighted version and real knockouts instead. Most of our Attention films and VR scenes (Attention I to X) were artistic, inspired by attention rather than drawn from it, and we say so wherever they appear; so are the two live pieces linked at the end. Attention XI, at the top of this section, is the exception: it draws Qwen3-1.7B’s real weights, sink and all.

TakeawayAn attention map shows where a model looks, not what it uses. To know what it uses, change the input and measure the output.

Reasoning trees I: the garden of forking paths

Every reply a model writes is one path through a huge branching tree of possible replies. To see that tree for real, we asked Qwen3.8-27B one question 3,072 times: “What would you want to tell someone who is afraid? Answer in one or two sentences.” We sampled at temperature 1.0 over its full vocabulary, which means drawing from its true odds with nothing trimmed.

All 3,072 replies are different. Stored as a tree of shared beginnings, they make 96,589 nodes. The average token is a mild choice (1.15 bits of uncertainty), but 21% of tokens are genuine forks above 2 bits. The tree grows faster than any list can follow: an exact search of the most likely beginnings covers 94% of the probability after one token, 52% after ten and 2% after twenty.

Grouped by meaning, the replies collapse into five ideas: you are stronger than you know (30%), fear is a signal, not a stop sign (19%), courage isn’t the absence of fear (17%), you have survived every hard day so far (17%) and you can handle what comes next (16%). Each idea is reached from 21 to 39 different first words. “Fear…”, “Your…”, “It…” and “You…” each lead into all five. The wording decides itself early; the idea is decided much later.

One more surprise. The reply the model gives if it always takes its single likeliest next word (“Fear is often a signal to pause and assess, not a command to stop…”) has a whole-sentence probability of about 1 in 1.7 million. Among the samples, the most likely complete reply was about 47 times more likely than that. Taking the best word at every step is not the same as writing the most likely sentence.

TakeawayWording diverges at once and meaning converges. Judge a model’s answer by the spread of ideas it produces, not by one sample, and don’t assume the default reply is its most typical one.

Reasoning trees II: search, honestly measured

A popular claim says that two models searching together can break through the plateau you hit when one model simply answers many times and you take the majority. We built a small, real version: Qwen3-1.7B and Qwen3.5-4B-Base write one line of working at a time into the same solution. Each candidate line is scored by a quick look-ahead (both models finish the solution from there), a yes/no check by the other model, and Python re-checking any arithmetic. The best paths survive each round. We then picked 40 maths word problems from the GSM8K test set that the 1.7B finds hard, and compared the search with each model answering 32 times and taking the most common answer.

The search lifted the weak model enormously, from 3 problems to 24, and beat simply pooling both models’ 64 blind tries (13). It did not beat the stronger model’s own majority vote (25), and it cost about 4.7 times as many tokens: 19.6k per problem against 4.2k. It beat both single-model majorities on one problem and lost to the 4B’s majority on two. The claimed breakthrough did not reproduce at this scale.

The reason is visible in the trees. The search never sees the right answer; it trusts its judges, and two small models are poor judges. The 1.7B says “yes” to nearly anything, and the look-ahead vote rewards whatever answer is popular. In problem #259, the true answer (168) was alive in round four with 59 look-ahead votes; the popular wrong answer (76) had 116, and by round five the right one was nearly gone. The search converged, confidently and wrongly. Meanwhile the right answer was usually somewhere in the samples: at least one of the 4B’s 32 tries was correct on 36 of the 40 problems. Finding a right answer was easy; recognising it was the hard part.

TakeawayA search is only as good as its judge. Agreement between two weak models is not verification, and the honest baseline (one model, many tries, majority vote) is hard to beat.

Long context: what holds over 32,000 tokens, and what breaks

Our earlier pieces never tested long inputs, so for this note we ran new measurements on Qwen3-1.7B. We took the public-domain text of The Wind in the Willows, cut it to lengths from 1,024 to 32,768 tokens, planted a made-up fact (“The secret number for the blue door is 7392.”) at different depths, and asked for it at the end. Without the fact, the model’s chance of guessing the number is about 2 in a million. We also recomputed the exact attention weights at the moment of answering.

Attention over a long text has a clear shape. 40% of all attention parks on the first token, the sink again, reaching about 70% in the last few layers. Another 27% goes to the last 15 tokens. Beyond that, the attention each token receives falls roughly in proportion to its distance. But there are so many distant tokens that, for a reader near the end of an 8,192-token text, everything more than 4,096 tokens back still gets about 6% of its attention in total, as much as the single previous token. A handful of retrieval heads do the long-range work. One (layer 21, head 14) puts 78% of its attention on the fact’s four digits from 6,147 tokens away. Another, in layer 7, starts at 86% on short texts but fades to 52% at 32,768 tokens, while the late heads hold steady. The logit lens shows the moment of recall: the lens reads a blank to fill in (“_____”) at layers 18 to 21, and the digit jumps to the top at layer 22, one layer after Rome did.

Simple recall turned out to be easy for this model. It recalled 330 of 330 planted facts at every length and depth. With the fact worded differently from the question (“The blue door opens only when you dial 7392”) it still found the right digits in all 150 runs, though in four of them it wanted to write “the” before the number. With up to 50 look-alike facts (“the silver gate”, “the red lamp”) in the text, it was exact 87 times out of 90. The three misses were all facts near the start, and in each the first digit came from a different fact: the one its retrieval heads were looking at most.

It broke on updates. We stated a number early, then changed it later in the text, and asked for the current number. When the outdated sentence used the question’s exact wording (“The secret number for the blue door is…”), the model favoured the old number in 12 of 15 runs. Swap which sentence matches the question and it gave the new number 15 times out of 15. With neutral wording on both, it was a coin flip (7 of 15). The model wasn’t tracking which fact was newer. It was finding text that matched the question and copying what came next.

TakeawayA small model can find one fact in 32,000 tokens reliably, but it finds it by matching wording, not by understanding the text. Needle-in-a-haystack tests mostly measure copying; tests with updates and conflicting facts are where long context really breaks.

What we got wrong along the way, and how we checked

Most of what we learned came from catching our own mistakes. These are the ones that would have ended up in a picture if we hadn’t checked.

• A river that rotated by itself. One of our layer-by-layer trajectory scenes re-fitted the 3-D projection at every layer, and the picture swung about 50° per layer. That motion came from our maths, not the model. One shared projection for all layers fixed it.
• A final layer that read 90.5% instead of 97%. The library hands back the last layer already normalised, and we normalised it again. Applying the step twice turned 97.0% into 90.5% in a recheck for this note. The lock-in layer, 21, was unaffected; the final number in that piece was not.
• Raw attention told the wrong story. Even with a dedicated start token added to absorb it, most attention still landed on the first real word. We switched to weighted attention and knockouts, and this note shows that even the weighted version is only loosely right.
• The lens is not the model. The logit lens reads noise until about layer 20 in the 1.7B and is close to useless in the middle of the 27B; a centred variant didn’t help. “The model knows nothing until layer 20” would be a wrong reading of our own figure.
• Brightness is not confidence. Our particle films add light where paths overlap. Going from 384 to 3,072 replies in Forking Paths blew the image out, so light has to be scaled down as more paths are drawn, or the brightest spot simply marks where the renderer stacked the most particles.
• A confident search flag. The search’s “verified” mark fired on all 40 problems. We now report it as “the search was confident”, never “proven right”, and the film’s problem was checked against the true answer.
• Picking the winner. The search film shows a problem where the search succeeded, chosen from six reruns. A rerun of #259 converged on a wrong answer, and it is in Figure 6 for that reason.
• A test that was too easy. Our first long-context test passed 330 of 330, because the question repeated the fact’s wording and copying was enough. It took three harder variants to find where the model breaks.
• Loose wording about our own work. River of Thought’s 4,624 tokens are sixty separate short passages, not one long context, and we first described it more loosely than that.

Methods and how to reproduce

Everything here comes from open-weight models running locally on one workstation GPU, read with forward hooks in Hugging Face Transformers and PyTorch. Layers are counted from 1 at the input (0 means the raw embeddings); heads are counted from 0. Probabilities are softmax outputs over the full vocabulary unless a figure says otherwise.

To reproduce the long-context results: load Qwen3-1.7B, take any long public-domain text, insert one sentence of the form above at a sentence boundary, append the question, and score the answer by teacher forcing (the probability of each answer token given the ones before it). Exact match means the model’s top token is right at every position, which is what greedy decoding would produce. For attention at long lengths, the full attention matrix is too large to store, so recompute softmax(q·k) only for the query positions you need, from the same queries and keys the model used.

Limits: the long-context and attention results come from one small model, on one book, with five to ten runs per cell. The 27B results are from single prompts. Idea clusters depend on the embedding model and on k. The search used small models and a modest budget; stronger judges would likely change the outcome. The pictures in the Lab pieces add their own layer of choices (layout, colour, timing), described in each piece’s notes.