Eiffel Tower Qwen

The paper

In Neural Networks Are Already Clean, Nick Cammarata of Goodfire looks at the individual "neurons" inside AI models. A neuron is a small unit inside the model: for every token the model reads or writes, each neuron produces one number, how strongly it is firing. Qwen3-8B has 442,368 of them.

The dominant view in the field of AI interpretability has been that most neurons are a jumble of unrelated jobs or concepts (called superposition).

Superposition meant that you couldn't identify one clear job/concept associated with a neuron. Instead a neuron might comprise a mix of totally unrelated concepts, making any further direct interpretation impossible without specialised tools.

This matters because it is the collective action of the neurons of an AI model that determine how it responds to human input. Change the neurons; change the output. If you could verifiably "read" a neuron then you could understand exactly why an AI came up with a given response.

The paper argues that (i) superposition is likely not correct and (ii) accordingly reading neurons is potentially much easier than commonly understood.

To support those arguments, the paper asserts two key findings:

Most neurons already do one readable job (rather than many). In Nick's audits of Qwen3-32B (a bigger version of the model we used), about 90% of neurons had one main job.
Bigger ideas are built from those readable parts. A concept is a combination of a few understandable neurons, not something hidden in a tangle.

If those two findings are correct then AI interpretability is, potentially, a significantly more tractable issue than previously thought.

We believe we have independently reproduced the core of both findings in Qwen3-8B (the first at least in part, see below), in a way anyone can check.

I have no background in ML or interpretability research, and relied heavily on Claude Opus 5.5 to run this experiment. We may be wrong, in aggregate or in the particulars, but we have checked that our results reproduce: re-running every concept from scratch found the same neurons. We also ran the same tests on a much smaller model, Qwen3-0.6B, which knows far fewer of our concepts but, where it does, again relies on a handful of neurons. On that model we also found the same key neurons again on a second computer, using freshly written questions (in 5 of 6 tries). More models may follow.

This site is intended to explain the experiment we have done and corroborate why the paper matters, in an accessible way. All the code will be open source on GitHub, to follow.

What we found

We used a small AI model called Qwen3-8B on a single RTX 5090. Other than the graphics card and Claude itself, we spent less than $20: API credits for the two AI models we used as graders (GPT-6-Luna and Jev). Claude Opus 5.5 wrote and ran the experiments; James set the questions and checked the results. In short:

A handful of neurons carry each concept. Switch off 5 or fewer neurons and the model's odds of giving the right answer fall at least 20-fold. That held in 157 of 158 tests across 139 concepts. Switching off random neurons does nothing.
Those neurons are readable. The Eiffel Tower runs through neurons for "landmarks", "buildings" and "Paris".
You can steer with them. Turn them up and the model starts talking about the concept, much like Anthropic's Golden Gate Claude.
You can do this directly from the neurons. Golden Gate Claude needed a sparse autoencoder to find its Golden Gate "feature". We needed no extra tooling: the neurons were already readable enough to find and use.

Confirmed in a test written down in advance. After all of this, we ran a final test on 40 concepts we had never used, with our predictions and scoring frozen before any data came in. Six of seven predictions held: the model recalled all 40 (and none of 4 new made-up ones); for 98% of them, 5 or fewer neurons carried the answer; and choosing neurons per concept steered 2.4 times better than a fixed rule. The one that failed was about people, and it failed in a good direction: they steered better than we predicted.

I don't claim these findings are unshakeable or perfect; only about 50% of the neurons we tested passed a strict one-job check, rather than the 90% the paper reports (this may be due to using a smaller model). Our "vibe-researched" approach no doubt has procedural issues or could be improved.

However, we do claim these results are reproducible, which we see as the key threshold.

We suggest the very fact that an absolute novice (albeit with an extremely capable LLM assistant) could reproduce key findings from the paper on prosumer hardware within 24 hours is strongly corroborative of the central assertion(s) of the paper: that AI/neural net interpretability is more tractable than many previously thought.

Figure 1 · Turn it up

We ask Qwen an everyday question while "turning up the dial" on the six neurons we identified as relating to the Eiffel Tower. Watch the tower creep into the answer, then take it over.

You ask:

Loading replies…

Eiffel dialoff

Figure 1. Six of the model's neurons linked to the Eiffel Tower, held at a multiple of the strongest they fire in ordinary web text. Every reply was generated in advance (the page runs no model); … marks where we stopped generating.

What's new here

Golden Gate Claude (Anthropic, 2024) and the Eiffel Tower Llama (David Louapre, 2025) both steered with a sparse autoencoder: a second model, trained separately, that translates the AI's insides into readable features.

To test the paper's key claims we instead automatically traced which of the model's own neurons push it towards each concept, and then tweaked those neurons.

As the figures show, changing those neurons predictably changes the behaviour of Qwen3-8B without requiring a sparse autoencoder (or other) interpretation aid. That is the development the paper points to.

Anthropic's more recent natural language autoencoders take the same route further, training copies of the model to put its internal state into words. Both are powerful, and both cost serious training; the paper's claim is that individual neurons are often readable without either.

Steering with a few raw neurons isn't new in itself (Suau et al. 2022 did it with about 3), and we aren't claiming a new idea. What we add is a confirmation of the paper's central point, that this is tractable: on a small language model, on consumer hardware, in a day.

Nor is it limited to a few showcase concepts: the same automatic method found a handful of key neurons for 157 of the 158 concept runs we tried.

Figure 2 · Switch it off

Now the reverse. We ask questions whose answer is the Eiffel Tower and switch off its most important neurons, one more at a time.

Show the control: switch off random neurons instead

Question, and Qwen's answerChance it's right

Neurons off0

Figure 2. Left: Qwen's answers to five of the Eiffel Tower questions, with that many neurons set to zero. (We left out one question that Qwen answers wrongly, as the Burj Khalifa, even with nothing switched off.) Right: the chance it answers "Eiffel Tower", averaged over all 16 of its questions (brown), against the same number of random neurons from the same layers (grey).

What Figure 2 shows: the model relies on a handful of specific neurons to name the Eiffel Tower. Switch off three and its odds of saying "Eiffel Tower" fall more than 20-fold; switch off twenty and it almost never answers "Eiffel Tower".

Notice what it says instead: Montmartre, the Arc de Triomphe, the Seine. It still knows the answer is in Paris; it has lost the tower. Switching off the same number of random neurons changes nothing.

Is the Eiffel Tower a lucky example? Figure 3 has every concept we tested, from coffee to justice, including the ones where it didn't work.

Figure 3 · Try any concept we tested

Concept

You ask:

Dialoff

Figure 3. Every concept we ran, each with the neurons our method picked for it automatically. Turn it up: steering on 8 everyday questions that were not used to pick the neurons. Switch it off: the concept's first three recall questions with its top neurons set to zero, as in Figure 2, with random neurons as a control. Right: the chance of the right answer as neurons are switched off, averaged over all the concept's questions, and what each steered neuron pushes the model to write next. Concepts that failed are listed at the bottom of the menu with the reason.

Built from shared parts

If concepts are built from readable parts, related concepts should share parts. We compared the top 50 neurons of 10 related pairs against 200 random pairs, after removing 18 general-purpose neurons that show up for almost everything.

Figure 4. Related concepts ("coffee" and "tea") share far more of their specific neurons than random pairs of concepts do: 2.6 on average against 0.13, about 20 times more. That suggests that, in this model, neurons combine to make concepts, and related concepts reuse each other's parts. None of the shared neurons is simply detecting the same word. Not every pair shares: the Eiffel Tower and Paris share none.

Where in the model

The model processes text in 36 layers, one after another. On the basis of the paper we hypothesised that we could infer (some) functions of layers from reading the individual neurons.

In testing this held: the concept's own neurons appear to cluster early (recognising what's being asked) and in the middle-to-late layers (the concept's parts), with a shared set of "say the answer" neurons at the very end.

This is not a novel discovery (see further reading), but it confirms that work and, the key point, it is directly traceable using the approach laid out in the paper.

It also illustrates a broader point: once individual neurons are readable, you can start to say what (some of) the layers of a particular model are doing (in some respects).

Turn the dial to see where the six Eiffel Tower neurons from Figure 1 sit in the model. Hover over or tap one to see what it does.

Eiffel dialoff

Figure 5. Qwen3-8B's 36 layers, from first (top) to last (bottom), each with 12,288 neurons (one dot here stands for 256). The brackets show what each band of layers does unusually often for the concepts we tested; every band does some of everything. The large dots are the six Eiffel Tower neurons from Figure 1, at their real positions.

How clean is the whole model?

We also checked 1,000 neurons picked at random to see how many have one clear job. With a strict check, about half did. The paper found about 90% in a model four times bigger. The details are in the methodology below.

What we haven't shown

This was a fast test, not a proof. The main gaps:

Superposition. We didn't test it head-on, so we can't rule out neurons packing in unrelated jobs we didn't see.
Needed isn't stored. Switching neurons off shows the model needs them. It doesn't show the fact lives there.
One main model. Everything on this page is Qwen3-8B. We also ran the same tests on the much smaller Qwen3-0.6B: it recalls far fewer of our concepts (7 of 60), but where it does, a handful of neurons again carry each one (mostly the same ones when we re-ran it on a second computer with fresh questions), and steering is weaker. That fits the paper's suggestion that bigger models are cleaner, but two sizes can't show a trend.
Attention. Each layer of the model has two parts: the neurons we studied, and "attention", which lets the model look back at earlier words. We didn't look at attention at all. Other work suggests it matters a lot for recalling facts (Choe et al. 2025 found that, in Qwen models, early attention matters more than the neurons for recall), so the neurons we found are part of the story, not all of it.
AI graders. Neuron descriptions come from GPT-6-Luna, checked by Jev. The steering scores we used to choose each concept's neurons come from a free local check that is a little generous. See the methodology.
Steering is modest. On fresh questions the concept usually slips into the reply rather than taking it over. People steer well when the question invites a person: in our final confirmatory test, Cleopatra, Abraham Lincoln, Gandhi and Martin Luther King Jr. turned up, fluently, in 43% of steered replies and never in unsteered ones. Some individuals still resist, notably Shakespeare, Mozart and Harry Potter. Our (untested) hypothesis for those: post-training may have given the model a general caution around famous creators and their works, to avoid reproducing copyrighted material, which it may apply even to people like Shakespeare and Mozart whose works are long out of copyright. This is an area for further research.

How we did it

James read the paper and set the goal: could we reproduce Golden Gate Claude straight from neurons (not sparse auto-encoders or other downstream tooling), and test the paper's claims along the way?

Claude Opus 5.5 (running in Claude Code) wrote the code, ran the experiments overnight, logged every result and flagged its own mistakes. James reviewed and steered.

High-level methodology, for those who want the detail

Finding a concept's neurons

For each concept, an AI (GPT-6-Luna) wrote 20 questions whose answer is the concept, such as "What is the name of the wrought-iron monument erected for the 1889 World's Fair in France's capital?". We kept the questions where Qwen gives the right answer at least a 20% chance. A concept counts as recalled if at least 5 questions survive.

Then, for every one of the 442,368 neurons, we measured how much it pushed the model towards the right answer on those questions. Technically, that is the neuron's activity multiplied by how sensitive the answer is to it ("attribution"). We rank the neurons by that score.

Finally we re-check the top 200 by switching each one off on its own and measuring the damage, and re-rank them by that. This matters: the attribution score alone sometimes puts the wrong neurons first.

Switching neurons off

We set the chosen neurons to zero, ask the questions again and measure how likely the model now is to give the right answer. We do the same with the same number of random neurons from the same layers, as a comparison.

We report the result as a change in the odds. "Odds fall 20-fold" means, for example, that a model 95% sure of "Eiffel Tower" (odds of 19 to 1) drops to about 49% (odds of roughly 1 to 1). We use odds rather than plain percentages because a percentage barely moves when a model starts out 99.8% sure, even when the damage is large.

Steering

Each neuron normally fires within a range. For every neuron we recorded the strongest it fires anywhere in about 2 million tokens of ordinary web text. Steering adds a multiple of that to the neuron's activity: at ×1 it gets an extra push as big as its strongest natural firing, at ×2 twice that.

We only do this while the model writes its reply; it reads the question normally. Figure 1 is slightly different: there the six neurons are held at that level rather than pushed up by it, the setting of our first run.

Each concept's neurons (2 or 4 of its top neurons) and strength were picked automatically on a set of practice questions, from three simple rules: the top neurons in the second half of the model, the top neurons anywhere, or the neurons that most directly push the answer word. Figure 3 then uses 8 different questions that played no part in that choice. Replies are greedy: the model always picks its single most likely next word, so the same question always gets the same reply.

Reading neurons

For all 442,368 neurons we stored the 40 passages where each fires most strongly, from 10 million tokens of web text. GPT-6-Luna read them and described the neuron's job. Jev then checked whether that description fits the passages, and also checked it against random text, to catch descriptions vague enough to fit anything.

The census of 1,000 neurons

The describer alone says about 81% have one job. But it also says that about half the deliberately mixed "two-job" neurons we planted as a test, so it's too generous. This is an area we want to optimise further.

With a stricter check, which lets through only 8% of those planted mixtures, about 50% of positive ends and 47% of negative ends pass (each neuron has two "ends", strongly positive and strongly negative, judged separately). That's closer to a floor than an estimate.

So our number is somewhere between a half and four-fifths, on a model a quarter the size of the paper's. That's below 90%. We didn't test whether bigger models are cleaner.

Controls and checks

Random neurons. Every switch-off and steering test is repeated with the same number of neurons picked at random from the same layers. If our neurons mattered only because we switched some neurons off, random ones would do the same. They don't: switching off random neurons leaves the answers unchanged, and steering with them never brings the concept into the reply.
Made-up concepts. We invented seven things that don't exist and described them to the question-writing AI as if they were real: Mount Velkaran (a supposed 6,214 m peak in Peru), the Draxwell Bridge (a supposed 1912 bridge in Ohio) and Elsabeth Quorrin (a supposed Dutch linguist), plus four more for the final confirmatory test: Mount Terenva (in Chile), the Orlen Reach Bridge, Dr Nima Vey (a materials scientist) and the Lantern Tide Festival. A method that found neurons for them would be finding noise. The model correctly failed to recall all seven, and the Draxwell Bridge caught one of our own steps inventing recall that wasn't there.
Planted two-job neurons. To test the neuron describer, we spliced together the examples of two unrelated neurons and asked it to describe the mixture. A good describer should say "two jobs". This is how we found that the describer on its own is too generous.
Predictions written down in advance. For four tests (the first 20-concept run, the layer test, the census, and a final confirmatory run on 40 new concepts) we wrote down what we expected before running them, so we couldn't move the goalposts afterwards. In the confirmatory run, six of seven predictions held. The seventh said people would steer, but less well than other concepts; they steered better (43% against 22%), so we report it as failed. That prediction was also badly set up: it compared people on questions that invite a person with other concepts on everyday questions.
Re-running from scratch. For this page we re-ran every concept. Where we still had the original neuron lists (59 concepts), the top 10 neurons came out identical, and the number of neurons needed to cut the odds 20-fold matched in 137 of 138 runs.

Code and full results: on GitHub, to follow.

[Eiffel Tower] [Qwen]