AI May Be Less of a Black Box Than We Thought

Goodfire argues that much of AI's inner working may be easier to examine than researchers thought. I see a possible new source of assurance evidence. First, we need to understand what has actually been observed, what is still a claim, and who benefits if we believe it.

When an AI gives us a bad answer, how much of the working behind it could we inspect?

Nick Cammarata's essay, Neural Networks Are Already Clean, published by Goodfire on 15 October 2026, argues that researchers have mistaken much of the organisation inside AI for a jumble. If he is right, investigating it directly could be more productive than his field has assumed.^[1]

Goodfire builds and sells tools for examining how AI works internally. Cammarata previously worked on this problem at OpenAI, where he started a project to automate that investigation. He also co-authored early research that helped establish the explanation he now disputes.^[1][2] He knows the argument from the inside.

The field is called interpretability: trying to explain what the software is doing between receiving a question and producing an answer. Goodfire is making a research claim about that work. My interest is what it might eventually let a buyer, investigator or board check for themselves.

Start with a comma

One of Cammarata's examples concerns the feeling of being proved right about a bargain. You chose the cheaper option, and it worked better than the expensive one.^[1]

To understand what the researchers found, we need to look below the words on the screen.

The learned system behind a service such as ChatGPT is called a model. A language model works with text; an image model works with pictures. It turns an input into an output through arithmetic. During its development, a learning process repeatedly adjusts that arithmetic using examples. This is training. The people building it set up the process; they don't write a separate instruction for every thing it will recognise.

Imagine pausing the software as it works. Numbers are being multiplied, added and used in later calculations. One result helps produce another. Researchers call certain small calculations in that chain neurons. The name comes from the brain, but these are maths in software, with no thoughts of their own. Different designs use different recipes for doing the arithmetic.

A neuron's result for a particular picture or piece of text is a number. Researchers call that its activation; I'll call it a reading. By collecting the examples that produce strong readings, they can start asking what that calculation contributes. They also examine strong negative readings where the design allows them, sometimes before a later step cuts them off. A neuron does not arrive with a label telling us its job.

In the bargain example, the strongest readings appeared at commas. Taken alone, those commas looked unrelated. But Cammarata describes reading the sentences around them and finding the common situation: satisfaction that a bargain had beaten the dearer alternative. The model was placing information about the surrounding text at the punctuation.^[1]

The observation was a set of readings at commas. The explanation was that this neuron tracked a particular feeling about a successful purchase. Those are different things. An appealing explanation still needs testing.

Goodfire reports that most neurons examined in its preliminary checks of language models seemed to have one main job, often accompanied by smaller, related jobs. Some remained unexplained. Its work includes Qwen3-32B, whose internal numbers are published for others to download and examine. Such systems are called open-weight models.^[1]

This is already more specific than saying AI has hidden order. The claim is that its existing calculations can often be understood, without first replacing them with something else.

Why cats and cars changed the argument

The older explanation was that a model had too few neurons for everything it needed to learn. It therefore squeezed unrelated things into shared parts, provided those things rarely got in each other's way. This is the meaning of "superposition" that Cammarata challenges. Researchers also use that word in other ways, so his essay does not settle every argument made under that name.^[1]

If unrelated things share a part, looking at that part alone may tell you little. Much of the field responded by building a second learned system to separate the mixtures, then studying the approximation it produced. Cammarata argues that this extra step has often been solving the wrong problem.^[1]

The famous case is an early image model called InceptionV1. A particular neuron responded strongly to both cat faces and car fronts. An animal and a vehicle seemed a convincing example of unrelated things sharing a neuron. Cammarata helped publish that interpretation.^[1]

When he returned to the pictures, he proposed a common shape: a V. He then looked at connected neurons and found V-shaped flames, water jets from a fireboat and fox faces. Round O shapes pulled the arithmetic strongly in the opposite direction. Those observations supported a different reading: the neuron was distinguishing pointed shapes from round ones, a distinction other parts could use.^[1]

That does not mean the model confused a cat with a car. Many other calculations contribute to identifying the picture. A shared shape can be useful to both.

Nor is the wider claim built only on renaming this one neuron. Cammarata describes improving the tools used to inspect image models, retraining models and watching how their neurons developed. In his account, poor learning and flaws in the inspection tools explained much of the apparent disorder. What remained often grew through related steps. One language neuron, for example, responded to "iy", then "diy", then "do it yourself", which contains no "iy" at all.^[1]

His proposed explanation is that these related extensions can look arbitrary if you only see the end result. Follow how they formed, and the connections become understandable. That is a hypothesis about how the organisation develops, not a promise that each neuron has exactly one meaning.

Make the explanation earn its place

There is an obvious danger here: once you expect a common thread, you can invent one.

Cammarata's answer is to keep trying to break the explanation. Give the model new examples that should produce a strong reading, and close alternatives that shouldn't. Check which calculations feed the neuron and which use its result. Increase the reading and see what changes in the model's output. Switch it off and compare the result.^[1]

A connected group of calculations doing part of a job is called a circuit. Following those connections matters because the same plausible label can fit several quite different explanations. "Cats and cars" might predict familiar examples well, yet miss the shared shape that explains the connection to flames.

Goodfire also uses AI agents, software that uses a model to carry out a task through a series of steps. In one experiment, an agent investigates a neuron and writes an explanation. Another receives only that explanation and writes text intended to produce a strong reading. That puts a prediction at risk before the result is measured. It tests something useful, although Cammarata acknowledges that reaching a neuron's strongest response does not establish everything it does.^[1]

Leo Gao, an interpretability researcher, wants a tougher test. He agrees that researchers dismissed neurons too quickly, but argues that a tidy description of what makes one respond is insufficient. Could that description be turned into a replacement for the neuron, with the model still performing its job just as well? If an explanation cannot stand in for the thing it explains, it may leave out important work.^[3]

Gao also questions Goodfire's treatment of faint readings. Goodfire concentrates on the strong ones and argues that small readings near zero usually matter less. Gao's best guess is that removing small readings would substantially damage performance.^[1][3]

This is a substantive disagreement about the evidence required. The replacement test asks more of an explanation than recognising a common theme. Both are relevant if someone eventually wants to sell that explanation as assurance.

How far the evidence actually reaches

Goodfire calls its picture a working hypothesis and puts its confidence at about 85%. That is confidence in the broad explanation of how these systems develop, not a claim to have understood 85% of AI.^[1]

Cammarata acknowledges that his title overstates the evidence. Goodfire has not audited everything, and it has not begun full audits at the scale of the largest models. It has also made less progress on the machinery that gathers information from different places in the text and moves it to where later calculations use it. Researchers call this attention.^[1]

These gaps matter. Understanding a neuron, or even a connected group, is different from explaining all the work behind a complete answer.

Neel Nanda, who leads this field of research at Google DeepMind, regards the work as interesting but does not see a major change to the overall picture. He still considers superposition real.^[4][5] Uday Kamath has reported a small replication in spirit, reproducing some patterns on much smaller models.^[6] That is useful follow-up, with limited reach.

In the limited search undertaken for this article, I did not find an independent check at the scale of Goodfire's main language-model results. Its promised technical companion paper had also not been found published by 26 October 2026. Those are limits on the evidence available here, not claims that no further work exists.

And "clean" is easy to misunderstand. Here it concerns organisation inside the model. It says nothing by itself about whether the model is safe, unbiased or secure. You can understand how a mistake happens and still have a system that makes it.

My view: give assurance another way in

From here, this is my professional judgement.

The attraction is an investigation that can go beyond collecting bad answers. If researchers can trace the calculations that contributed to an error, they can test whether changing or removing those contributions changes the outcome. That could help establish a cause, alongside tests of how the system behaves from the outside.

I would welcome that as another kind of evidence. A supplier saying "we tested it and it worked" leaves different questions open from an investigation showing which parts did the work and how that explanation was checked.

It would complement behavioural testing and the controls around a model. An AI agent that can take actions in other software still needs limits on what it is authorised to do. Controls outside the model can enforce those limits even when the model produces the wrong instruction.

Reading the internal working is also different from reliably controlling it. A model update, further training for a particular task, or an attack can change the parts or how they are used. Evidence about one version and one set of circumstances should not quietly become a guarantee about the next.

Access is a serious practical constraint. Open-weight models let outsiders examine the internal calculations. Closed commercial services, including the models behind ChatGPT, do not give their customers that access. Goodfire's research therefore does not hand an organisation a way to inspect whichever AI service it already buys.

There is substantial specialist work involved too. Goodfire relies heavily on AI agents helping to investigate other AI. That makes scale possible, but their explanations and experiments need checking as well. I would not plan on a dashboard that makes this available to an ordinary buyer tomorrow.

Follow the financial interests too

The money gives me another reason to insist on evidence. This is my analysis of incentives, separate from Goodfire's scientific claim.

OpenAI announced a funding round in March with US$122 billion in committed capital at a US$852 billion valuation.^[7] Anthropic announced in February that it had raised US$30 billion at a US$380 billion valuation.^[8] Committed capital, money raised and the value placed on a company are distinct figures. None of those figures is an infrastructure spending commitment.

A business raising capital on that scale benefits from a story of extraordinary capability. There is also commercial value in a story that the product is so complex only its maker can properly vouch for it. It helps justify the spending and keeps the supplier in the position of principal authority.

That is a reason to seek evidence beyond the supplier's account. It is not evidence of deception or misconduct. It does not establish that the models are simpler than claimed or that the infrastructure is unnecessary. A model can be readable and still require enormous computing power.

Goodfire has interests of its own. Anthropic invested directly in its US$50 million funding round in April 2025.^[9] Menlo Ventures, which led that round, says it first backed Goodfire through the Anthology Fund that it co-leads with Anthropic, and described the later investment as Anthropic's first direct investment in another company.^[10]

Goodfire subsequently raised US$150 million at a US$1.25 billion valuation in February 2026.^[2] It sells interpretability tools. Success for the argument that AI can be understood is commercially useful to it.

The incentives do not all point towards emphasising complexity, either. A developer able to describe its model as readable gains something useful in conversations with regulators and risk committees.

I would therefore treat Goodfire as the author of a testable claim, with a commercial stake in the outcome. It cannot independently validate its own work. The public argument and the attempts to check it are what make this worth following.

What I would ask a supplier

I would start with the scope: has anyone examined how this particular model produces its answers, or has the work been confined to testing those answers?

Then I'd ask about the people and the money: who carried out the examination, who funded it, and how independent were they of the supplier?

And I would press for specifics behind the word "interpretable". Which version was examined? Which parts and behaviours were covered? What remained unexplained, and what would need checking again after a change?

Those questions do not require a board to become an AI research team. They require a claim to come with enough detail to judge its relevance.

I'm optimistic about the possibility of following more of the working. For now, informed interest is the useful position: keep the controls in place and make the people selling assurance show what supports it.