What this is
Nick Cammarata's essay argues that most neurons in a language model already do one readable job, and that mapping them should be routine work: the "normal science" of interpretability. This page is our attempt to engineer that. We treat it as an engineering problem more than a research one.
It comes down to two things.
- Label each part of a neuron separately. A neuron has three parts: the gate (what opens it), up (which side it takes) and the write (what it outputs). We label each one on its own.
- Make every label prove itself. Each label is checked against the model, with controls and fake neurons. A label that fails is shown as failed.
We think the useful part is the combination. Almost every ingredient has been published before (see Credit). Put together, using extremely fast and cheap models (gpt-6-luna and Jev), they label neurons cheaply enough to do it at scale, with intentional transparency as to how much each label can be trusted.
Written for people who work on ML and interpretability, with plain-English definitions on hover for everyone else. It's the gist, ahead of a full write-up, so we can get feedback first.
How we think about a neuron
Without any grounding in the field, one of the first points we sought (and seek) to confirm was that our understanding of neurons was right. This section seeks to explain our understanding in an accessible and visual way. In summary, our understanding is that every neuron in Qwen3-8B's MLP layers does one sum for each token it reads: activation = SiLU(gate) × up. It then adds that activation × a fixed down vector to the residual stream. We picture it as three parts.
- The gate is a castle gate with a ruler at the door that detects certain concepts only. On L31:10454 (a neuron related to bicycles) you could paraphrase it as "you must be at least this wheeled-ish to enter". It measures how much of that concept a word has, and opens further the more it has. Fall short and almost nothing gets through (the gate is soft, so a trace always leaks).
- Up picks a side. Once a word is through, a second reading says which way it goes, + or −. The two sides can have different triggers: on L31:10454, bicycles, scooters and motorbikes go one way; other ways of moving or exercising (wheelchairs, skis, treadmills, kayaks) go the other. The sides can also do separate jobs: a concept the model needs from the neuron uses one side or the other.
- The write is the neuron's output; here a ballista with two fixed targets. Up turns it towards + or −, and how far the bolt flies depends on both readings. Each target is a set of words that direction pushes the model towards.
Which models this fits. This picture applies to dense models with SiLU-gated MLP neurons, like Qwen3-8B.
Verbs, not nouns
We try to describe a neuron by what it does, not by what it's about. "The bike neuron" is a noun. What the neuron actually does is closer to: opens for anything wheeled; for bicycles, scooters and motorbikes, pushes one way; for other ways of moving or exercising (wheelchairs, skis, treadmills, kayaks), pushes the other way. That's a job.
A job is built from the neuron's three parts, each measured and labelled independently: the gate and up from text, the write from the weights.
- Gate and up: what opens it, and which side it takes. The gate's label, plus one label for each side of up. Checked by the exam, the generative check and the fake neurons. A neuron can be partly labelled, for example only one side of its up.
- Write: how each side's push lines up with what it read. From the weights alone, for all 442,368 neurons.
- Amplifier: pushes further towards what it read. Amplifiers dominate layers 6–17 (68–79%).
- Eraser: pushes against what it read. Never common: erasers peak at 13–14% in layers 18–23 and 30–35.
- Redirector: pushes somewhere unrelated. Layers 24–35 are mostly redirectors (79–86%), and so are layers 0–5 (69%).
The job is an abstraction from those labels and from knockouts: what the neuron is for, where and when it acts, judged from which concepts break when it is switched off.
- Jobs so far:
- Question-side readers (early layers): act on the question's words.
- Answer-side workers (late layers): act where the answer is produced, and only when recalling it.
- Middle layers: vary; some act on the question, many at the answer.
How the parts fit. The whole chain (the label agrees with what breaks, a route is found, and restoring it undoes damage) holds for about 6% of held-out testable neurons, most of them early question readers or late answer workers. We haven't shown that write type predicts the job.
Some neurons carry a flag rather than a job: hub (needed by many concepts), candidate junk, token-locked (fires on its token almost wherever it appears; 34% of word-shape labels, 7% of meaning labels) or unexplained. Still open: inhibitors acting as gates; saying a word versus spelling it; answers longer than one token; steering past the point of sense; routes through attention heads.
How we label each part
The gate and the two sides
- Vocabulary sweep. We put every one of Qwen's ~151,000 tokens into 12 short template sentences, such as "She spent the whole afternoon talking about [X] with her friends.", and record every neuron's gate and up readings in one pass. That gives each neuron its favourite tokens: by gate, by strongest + side and by strongest − side. Glitch tokens are dropped.
- Describe. An LLM (Luna) sees half of the favourites and writes a short description. It's allowed to say there's no pattern.
The write
The write's direction is fixed by the weights, so we read it directly: the words each direction pushes the model towards, through the model's output table (the logit lens), and its job type per side. Then we ask what breaks when the neuron is switched off: which concepts the model can no longer recall (knockouts).
How each label checks itself
- Label exam. A separate calibrated judge (Jev) gets only the description. It must pick the neuron's held-out favourites out from look-alike decoys. The score is compared against five other neurons' descriptions on the same test, so a vague label that fits anything doesn't pass.
- Fake neurons (the falsifier). We build pseudo-neurons from random mixtures of real gate rows and random directions, and run them through exactly the same pipeline. If the method "finds" jobs for them, it is reading noise.
- Generative check. Luna writes brand-new words from the label alone. The real neuron has to fire on them, more than on words written from a description of only the tokens' surface (spelling, script, length). This catches labels that only describe what the tokens look like.
- Real text (the corpus exam). A label is written from 12 template sentences, so we also test it on ordinary text it never saw: about 9.3 million tokens of web pages, chat questions and Wikipedia intros. The judge sees only the label and rates passages that end at a marked token: the neuron's 30 strongest moments, mixed with three kinds of decoy. The hardest is the same token in a passage where the neuron stays quiet. It holds the word fixed, so a label passes only if it captures the context that opens the gate. A neuron that fires on its token almost everywhere is token-locked: a different kind of label, not a fail.
- Knockouts (for the write). We switch the neuron off and measure which concepts the model loses, against random neurons from the same layer. Then we test whether the label predicted which concepts would break.
| Part | Pass the label exam | … and the generative check |
|---|
How far each label gets
The checks above get harder one after another, so each gate label can be placed on a ladder: how far down this list it gets. The table gives the share of gate labels that pass each step, and how many fake neurons do (they should almost never). It covers gates only, because the real-text steps haven't been run on the two sides yet. All eight predictions we set in advance for the real-text steps were met, one at the edge (a keyword baseline of 0.599 against a bar of 0.60).
| Step | What the label has to do | All gate labels | Of those that passed the exam | Fake neurons |
|---|---|---|---|---|
| 1. Exam | Pick the neuron's held-out favourite tokens out from look-alike decoys | 68% | all, by definition | about 2% |
| 2. Generative check | Words written from the label alone make the neuron fire | 54% | 79% | about 1% |
| 3. Real text | Pick the neuron's strongest moments in ordinary text out from other neurons' | 71% | 86% | 4.5% |
| 4. Same word | Pick them out from the same word where the neuron stays quiet (or the neuron is token-locked) | 64% | 81% | 2.2% |
| 5. Same topic | Pick them out from strong moments of neurons with similar labels | 31% | 37% | 2.5% |
Steps 1 and 2 cover the 11,000 neurons on the map; steps 3–5 all 12,244 labelled neurons, both of today's runs and two earlier ones. Fakes: all 200 fake neurons. Other neurons' labels score 0.49–0.51 on every real-text test, which is chance, and a judge-free keyword match scores 0.60 on same-word decoys, so a label can't pass just by naming the word.
A graded check: simulation (every label; set in advance, all bars met). The steps above are pass / fail. In simulation the judge reads only the label and guesses how strongly the neuron fires at 60 positions in web text, from its strongest moments down to ordinary ones, and we score how well the guesses rank with the real values. Verified labels reach a median rank correlation of 0.65 for gates and 0.55 for sides, against about 0.1 for other neurons' labels and 0.1–0.2 for fake neurons. Each neuron's own score is in its pane on the map.
Beyond the ladder: what breaks. A label can also be checked against what breaks when the neuron is switched off. Labels that pass the same-word test agree clearly with the knockouts in every layer band (+0.17 to +0.28); labels that fail it barely do (+0.02 to +0.04). This needs knockout data: 10,359 of the 11,000 neurons on the map have it, since the knockout-linked set was chosen because knockouts tie them to concepts. These agreement figures come from the 843 held-out and knockout-tested neurons the comparison was run on.
Limits. The write is the least verified part. Labels agree with which concepts break, in early and late layers, but they don't predict new breaks. The whole chain (label, write, the later neurons it feeds, and a check that restoring them undoes the damage) holds for 6% of held-out testable neurons. The labels themselves hold up against look-alike decoys and on real text, but are blunt within their topic. More in Findings; every part's verdict is shown, including the ones that fail.
The map: 11,000 neurons
11,000 neurons from all 36 layers. Each bar is one layer, split by how many of a neuron's parts are labelled. Click a layer to see its neurons. Hover over a square to read a neuron; click to pin it.
Why it's cheap and fast
We read neurons directly, so there is nothing to train. Sparse autoencoders and transcoders first train a second model to rewrite the network's activity into readable features, and those features then still need labels. Here the units are the model's own neurons, and labelling them is one sweep plus API calls.
- The setup: one desktop with an NVIDIA RTX 5090 (32 GB) for everything that runs the model; OpenAI's
gpt-6-lunaas the labeller that writes descriptions and words; TypeSafe's Jev (jev-latest) as the judge. - 11,000 neurons, every part: about 2 hours 41 minutes and $85 in API calls.
- Checking every label on real text: $14.81 of judge calls for 12,244 neurons (about $0.0012 each), plus one 22-minute GPU pass over 9.3 million tokens. Each new batch of labelled neurons needs its own pass of about 20 minutes.
- The sweep: a single pass over the vocabulary, about 20 minutes for all 442,368 neurons on one RTX 5090.
- The labelling: parallel API calls, and the judge answers about 9,000 questions a second. So labelling scales with API throughput, not GPU time.
- The whole model: at the same cost per neuron, labelling all 442,368 neurons would take about $3,400 of API calls and one sweep. That is an estimate from the runs above, not a run, and covers labelling only, not knockouts. Checking those labels on real text would add about $530, with the GPU pass planned over all neurons at once.
- In comparison to Transluce's neuron-description pipeline: about $0.009 against $0.046 per neuron (whole model about $3,900 extrapolated against about $21,000) and roughly 2% of GPU time. Their pipeline scores descriptions by simulation. Ours scored about 36,600 labels for $36, about $0.001 per label, after one 73-second GPU pass that stored every neuron's values at 4,000 web positions; the whole model would be about $1,300 with no new GPU pass, against Transluce's $5,174 and 2,891 GPU-hours. The scores themselves aren't comparable: different positions, rank correlation, our judge, another model.
The trade-off is coverage. A neuron basis won't capture everything a learned feature basis can, and about 11% of the 11,000 are still wholly unexplained.
Findings
All on Qwen3-8B, from the runs to 21 October 2026. Most of it is exploratory; where a check was set in advance and failed, we say so.
What switching neurons off shows
Our earlier work in this direction, concept by concept, is on the concept map.
- Questions. Luna writes about 10 questions whose answer is the concept's name, without giving it away ("Which iron lattice tower was built for the 1889 World's Fair?"). We keep only questions Qwen answers (at least 20% on the name), and a concept needs at least 5.
- Shortlist. A fast gradient estimate (gradient × activation) picks 200 candidates out of all 442,368 neurons.
- Switch off one at a time. Each candidate's activation is set to zero at every token. Its effect is the drop in the log-odds of the right answer, averaged over the questions. The top 30 are the concept's map.
- Switch off together. The top 1, 2, 3 … 50 are zeroed together, against random neurons from the same layers, to see how many it takes.
- A handful of neurons are each necessary. For 99.0% of 4,194 concepts, switching off 5 or fewer neurons cuts the odds of the right answer about 20-fold (a log-odds drop of 3 or more). With 5 or fewer switched off, the right answer drops below even odds on at least half of the questions the model was getting right, for about 92.7% of concepts.
- But the answer draws on dozens. Keeping only a concept's top 30 neurons working saves just 28% of what its 200 tested neurons do together; saving half takes a median of 61–80. A median of 32 tested neurons matter on their own (a drop of more than 0.30), and that is a lower bound.
- Across the 4,194 concepts, 29,039 distinct neurons are reliably needed by at least one (about 7% of the model); 46,095 (about 10%) pass the looser 0.30 bar. They are fairly specific: a mapped neuron breaks concepts it was never flagged for in about 1.25% of pairs, against 0.43% for random neurons.
- The last layers write the answer. About two thirds of each concept's top 5 neurons sit in layers 28–35. The search favours the layers that write the answer, so early neurons are under-counted.
What the labels show
- Each part of a neuron can be labelled on its own. Across 11,000 neurons, 68% of gates and about 45% of each side get a label that passes the exam. Fake neurons pass about 2% of the time on the exam alone, and about 1% with the generative check too. As a further check, we ran the same pipeline on a copy of Qwen3-8B with its layers' weights randomised, so that no neuron should mean anything: 0 of 294 of its neurons got a verified label.
- Labels hold up on real text. 71% of all gate labels pick out the neuron's strongest moments in ordinary text, and 64% still do when the decoy is the same word in a context where the neuron stays quiet (86% and 81% of labels that passed the exam), against 4.5% and 2.2% of fake neurons.
- The middle layers are the hardest to label, in every full run and on real text (74% on same-token decoys, against 87% early and 81% late): gates pass 38% in layers 12–23 against 45–54% elsewhere on random neurons, and 57% against 72–74% on knockout-linked ones.
- The two sides are labelled separately. They can respond to different things; on 61–64% of neurons both sides have the same write type (above).
- Few neurons look like junk. About 4–4.5% of neurons tested by knockout (on the concepts the search had flagged for them) are candidate junk: no part can be named and switching them off has no effect above a null-calibrated bar.
Limits of the labels. They are reliable against look-alike decoys, not against meaning-matched ones: with near-synonyms as decoys, the pass rate falls from 57% to 27%. On real text, only 37% of passing labels can tell the neuron's strongest moments from moderate members of its own topic. The exam doesn't penalise a label that is too broad or too narrow. And the generative check is not neuron-specific on its own: fake neurons pass it 16–25% of the time, so it counts only alongside the exam (about 1% of fakes pass both). Finally, on real text a label describes a neuron's peaks best: without its strongest moments it still tracks part of the everyday firing (0.42 for gates, 0.24 for sides), but much of the weaker activity is not captured by the label. That is where polysemy and weak activations would sit, and it is an open question.
Do labels match what knockouts break? (exploratory)
- Many "mixed" knockout lists are two one-sided jobs. Each concept a neuron is needed for uses one sign of the neuron consistently (94% of 18,000 neuron–concept pairs), about half of them the negative side, and concepts that share a sign are more alike than those that don't. Switching a neuron off removes both sides at once, so its list of broken concepts mixes the two (an example). Matching each concept to the side label it uses helps predict the list, clearly in early layers, modestly elsewhere. (Set in advance; two of three predictions met, the third in direction only; replicated on a second sample. How.)
- Labels that hold up on real text agree with knockouts, in every layer band. Where a neuron's label passes the same-token test, its label matches the concepts it breaks clearly better than other neurons' labels (+0.17 to +0.28); where it fails, barely (+0.02 to +0.04). That includes the middle layers, so their weak agreement looks mostly like weak middle labels. Two independent checks, real-text firing and which concepts break, line up neuron by neuron.
- Labels agree with which concepts break, but don't predict new breaks. A neuron's own label matches its broken concepts better than other neurons' labels do (+0.19). On 400 held-out neurons this holds in early layers (+0.23) and late layers (+0.14), not the middle (+0.04). In early layers it is largely because the broken concepts' questions contain the neuron's trigger words. For concepts the original search never flagged, a label's predictions rarely come true: 0.3% for random neurons and 2.7–4.6% for strongly mapped ones, against 0% for controls.
How a neuron's effect reaches the answer: routes (exploratory)
A route is a link read straight from the weights: neuron A's write lines up with what a later neuron B reads, and B is needed by a concept that switching A off breaks. Reading routes needs no GPU and no training (the "virtual weights" idea of Elhage et al. 2021), which is part of why this is cheap.
- The weights predict routes. A neuron's write lines up better with later neurons needed by the concepts it breaks than with those needed by concepts it doesn't (AUC about 0.71; 0.72–0.74 on 400 held-out neurons). Another neuron's write from the same layer scores about 0.53, and random directions score at chance. About a third of tested neurons have a significant route.
- Only early routes are clearly the neuron's own. To test whether a route belongs to the neuron or just to its subject, we compare it with the write of another neuron on the same topic (one needed by similar concepts). In early layers the neuron's own write lines up with the later neurons clearly better: by +0.16 to +0.31 in AUC, across three samples. In middle and late layers the other neuron's write does nearly as well, so the route is partly shared by neurons on the same topic rather than specific to one.
- Knockouts confirm some of them. Switch A off, then restore B to its normal activation: a median 16% of the damage is undone, more than three matched controls. Restoring all of the concept's later needed neurons undoes about half. This holds on 300 held-out neurons.
- Most confirmed routes are direct. In 64% of held-out early and middle routes, most of the effect runs from neuron to neuron at the same token position, and restoring only A's direct write into B does about as much as restoring B.
Limits. A single route can flip between two GPU types (about 1 in 5); pooled results agree. The whole chain (the label agrees, a route is found, restoring it undoes damage) holds for 6.0% of 400 held-out testable neurons (95% interval 3.8–8.5%; 5.0% counting only results stable across GPUs). Routes that cross token positions go through attention heads, which we haven't mapped (next).
Where a neuron's effect enters, and what attention carries
A neuron only acts at the token it is reading. To reach an answer that comes later, its effect has to be carried forward by attention. We tested where each neuron's effect enters by switching it off only at the question's tokens, or only where the answer is produced.
- Early neurons act on the question. A median 96% of their effect comes from the question side (98% on fresh neurons). (replicated on fresh neurons) The question's words carry a median 94% of the total effect and the chat-template tokens 2%, so these are question readers, not turn-boundary neurons. (set in advance) The answer comes later, so attention must carry their effect forward.
- Late neurons act at the answer. Only 8% of their effect comes from earlier positions. (replicated on fresh neurons)
- Middle neurons vary. Some act on the question, many at the answer, and the mix differs between samples.
- Late answer neurons are recall-specific. This is our reading of a test set in advance that failed: they barely matter when the concept is named in the question (an effect ratio of about 0.0002), or when the model only repeats the name (0.000 across 100 prompts, all recalled). (replicated twice)
- No general "name reader". Our reading of a test set in advance that failed: early neurons that matter at the concept's name are mostly tied to particular facts, and a third independent split missed its bar (0.087 against 0.15). A search at the name position puts 87% of the top 5 neurons in layers 0–12, partly by design.
What this means for the method. Our labels and routes are at the level of MLP neurons. Most confirmed routes are direct, at the same position (above). Effects that cross positions travel through attention, which we haven't mapped: there are no head-level routes yet. Why the middle layers are hardest is not settled, but the corpus exam points to weaker labels: middle neurons whose labels pass on real text agree with knockouts (+0.17), and those that fail don't (+0.02). Effects carried by attention, which we don't map, may add to it. Where effects enter was first found exploring, then rechecked on fresh neurons as marked. The runs already save attention summaries at the answer position, not yet analysed.
Code
Everything on this page can be rerun with Model Mapper 0.2.0. modelmapper label labels any neurons, layers or a whole model, with every check above, and shows its cost before it starts. modelmapper check --labels tests labelling on your own machine against our values. modelmapper lookup shows the labels of all 11,000 neurons on the map, offline. modelmapper realtext runs the corpus exam and simulation; it is the research code, packaged as experimental. Numbers here are from the runs of 21 October 2026.
Credit and prior work
Nick Cammarata's essay was the main inspiration for this work, and all of it was done after he published it. Any omissions of prior art are unintended; James has no formal background in the field and most of the methods used were found via iteration with Claude without James having conducted any literature review. Subsequently Claude's literature review confirmed many of the methods had been identified previously. It seems likely that much of that work will be in Claude's training data, so may well have (inadvertently) shaped the approach. Our preference at this stage is therefore to credit wherever we are aware that credit is plausibly due.
- Neural Networks Are Already Clean, Cammarata (Goodfire). The argument this page follows on from.
- Bills et al. 2023, Language models can explain neurons in language models; Paulo et al. 2024, Automatically interpreting millions of features; Transluce neuron descriptions. Describe-then-score with an LLM: our label exam has the same structure.
- Paulo & Belrose 2025, Evaluating SAE interpretability without explanations. Scoring the examples themselves, as our list exam does.
- Choi, Huang, Meng, Johnson, Steinhardt & Schwettmann 2024, Scaling Automatic Neuron Description (Transluce). Describing every neuron of an 8B model; the cost comparison in "Why it's cheap and fast".
- Bau et al. 2017, Network Dissection; SAEBench 2025. Comparing real units against random ones: the idea behind our fake neurons.
- Singh et al. 2023, SASC. Running every input through a module and summarising the strongest: close to our vocabulary sweep.
- Shazeer 2020, GLU Variants Improve Transformer; Shabalin & Piotrowski 2026, Comparing Sparsified MLPs and Transcoders: Many Transcoder Features Are Neurons in Disguise. The gated MLP, and splitting a neuron into two sides by sign.
- Gerstner & Schütze 2025, arXiv 2505.17936. Amplifier and eraser neurons from the weights, with the same layer pattern (they call it enrichment and depletion).
- Geva et al. 2021, Feed-forward layers are key-value memories; Geva et al. 2022, promoting concepts in the vocabulary space. Reading a neuron's output as the words it pushes.
- Elhage et al. 2021, A Mathematical Framework for Transformer Circuits (virtual weights); Meng et al. 2022, ROME (causal tracing); Wang et al. 2022, path patching; Dunefsky et al. 2024, transcoders; Anthropic 2025, attribution graphs. Write routes, knockouts and mediation.
- Choe et al. 2025, arXiv 2509.08778. Early attention matters for recall in Qwen; we don't label attention here.