The claim arrived as a single sentence. "Most AI neurons are now interpretable." No paper. No model card. No layer range. No evaluation protocol. No timestamp on the original post. It was attributed to a former OpenAI researcher, Nick Cammarata, and relayed by a crypto asset outlet whose editorial beat is tokens, not transformer internals.
I have audited smart contracts that shipped with more documentation than this. At least a contract has a compiler. At least a contract fails loudly when the logic is wrong. This claim has neither property. It cannot compile and it cannot fail, because it was never specified precisely enough to be wrong. That is the finding. Not that the claim is false โ that it is unfalsifiable as published. And the framing, neuron-level interpretability, achieved, now, runs against the direction of every credible result in the field over the last three years.
Check the inputs, ignore the hype. The input here is one sentence from a secondary source. The hype is the word "now." That word is doing all the work. It implies a threshold was crossed. Thresholds have dates, metrics, and reproducibility. This one has a headline.
Interpretability was not always a press release. Between 2020 and 2022 it was an artisanal discipline โ researchers stared at activation patterns, drew heat maps, and wrote anecdotes about "the cat neuron" or "the sentiment direction." The work was real but it did not compose. Every finding was local, model-specific, and unreproducible in the way that matters: you could not take a method that explained one layer of one model and apply it to another.
The field's response, from 2023 onward, was to industrialize. Anthropic published Towards Monosemanticity in May 2023, applying sparse autoencoders to a small transformer to decompose activations into features that fired more cleanly than raw neurons. In May 2024, Scaling Monosemanticity pushed the same technique onto a production-grade model and reported tens of millions of extracted features โ the figure usually cited is roughly 34 million on the Sonnet-class model. OpenAI followed in June 2024 with Scaling and evaluating sparse autoencoders, extending the approach to a frontier model and, critically, publishing evaluation methodology alongside it. DeepMind shipped Gemma Scope in August 2024, releasing SAEs for an open model so that outside researchers could reproduce and attack the claims.
Notice the shape of that progression. Each step added scale, and each step added evaluation discipline. The field did not declare victory; it built the scaffolding required to even argue about victory. The vocabulary shifted deliberately: from neurons to features, from interpretable to measurably interpretable under a stated protocol. That shift was not aesthetic. It was forced by a technical fact that the headline under review ignores entirely.
The fact is polysemanticity. Individual neurons in a trained network are not clean concepts. They fire across multiple unrelated inputs โ a single neuron responding to legal terminology, academic citations, and the color red, depending on context. This is not an occasional defect; it is the default. The reason SAEs became the consensus tool is precisely that neuron-level interpretation kept failing. The neuron was abandoned as the unit of analysis because it did not survive contact with the data. Any claim that "neurons are now interpretable" must therefore either overturn polysemanticity or redefine what a neuron is. It does neither.
Start with the word itself. "Interpretable" is the Achilles heel of the entire claim, and it is never defined. In the literature it carries at least four distinct meanings, and they are not interchangeable. It can mean a human-readable natural-language description of what a unit does. It can mean a causal explanation that predicts how the model's behavior changes when you intervene on that unit. It can mean an explanation that an automated system can verify against held-out behavior. Or it can mean mere separability on a specific probe task โ the unit distinguishes class A from class B in a controlled setting.
These are different claims with different evidence requirements. A natural-language description can be fluent and still wrong. A probe result can be clean and still not reflect the mechanism the model actually uses. Conflating them is how a research nuance becomes a headline. The article under review makes no distinction, which means it cannot be graded. A claim that cannot be graded is not a claim. It is a mood.
The strongest counter-evidence comes from inside OpenAI itself, which is awkward given the headline trades on OpenAI credentials. In 2023, Bills and colleagues published Language models can explain neurons in language models, using GPT-4 to generate natural-language explanations of GPT-2 neurons and scoring those explanations by how well they predicted activation on held-out text. The headline finding is often quoted as a success. The load-bearing finding is the divergence: explanations that scored well on the explanation metric frequently failed to correspond to the actual causal mechanism when tested by simulation and intervention. High explanation score, low simulation fidelity. The two metrics disagreed, and the disagreement is the point.
That result is a direct refutation of the reading the headline invites. If automated natural-language explanations โ the most generous interpretation of "interpretable" โ can score well while misrepresenting the mechanism, then "interpretable" cannot mean "we can describe it." It must mean "we can predict and control it." And prediction-and-control is exactly what no published result has demonstrated at neuron granularity across a large model. The code was solid; the logic was not. The tooling works. The inference drawn from it does not.
Here is what a defensible version of the claim would require. First, a named model: architecture, parameter count, training data provenance. Second, a layer range: interpretability is not uniform across depth, and a result on early layers does not generalize to the residual stream near the output. Third, an operational definition of "interpretable" tied to a measurable task. Fourth, an evaluation protocol with inter-rater agreement if humans are judging, or a held-out benchmark if a model is judging. Fifth, a causal test: intervene on the unit, measure whether behavior shifts in the direction the explanation predicts. Sixth, independent replication by a party with no stake in the result.
The article provides none of these. Not one. It provides a name, an institutional affiliation that is no longer current, and an absolute quantifier โ "most." In risk terms, this is a claim with no control sample, no confidence interval, and no audit trail. I would reject it in a due-diligence memo without a second read.
The propagation chain matters as much as the content. The original source is almost certainly a social media post โ a short-form observation, likely qualified with phrases like "in some layers," "by one metric," or "for a subset of features." Those qualifiers are the load-bearing structure of the claim. They are also the first thing lost when a short post becomes a headline. "Most neurons are now interpretable" is what remains after every hedge has been stripped. This is not malice. It is compression. But compression that deletes the qualifiers deletes the science.
I have watched this exact failure mode in crypto. A researcher posts a thread noting that a specific contract under specific conditions exhibits a specific behavior. Within a day it becomes "protocol X is exploitable." The thread was right. The headline was wrong. The distance between them was three removed clauses. Silence in the logs speaks louder than bugs โ and here the silence is the missing methodology, the missing timestamp, the missing replication. What is absent tells you more than what is present.
There is a commercial subtext the article never touches, and its absence is diagnostic. The interpretability toolchain โ SAE training pipelines, feature browsers, probe libraries, the startups built around them โ derives its value from the premise that features, not neurons, are the correct unit. If neuron-level interpretation were genuinely sufficient, that premise weakens. The differentiation of every SAE-based product narrows. A serious analysis would flag this tension: the claim, if true, is a partial bear case for the very infrastructure the surrounding industry is funding. The article does not flag it. That tells me the author has no map of the field's incentive structure. They are relaying a signal, not reading it.
Now the part that actually concerns me, because I have been on the wrong side of an automated system making decisions faster than a human could audit it. In 2025 I analyzed an AI-driven trading agent protocol. The agents were sophisticated, well-documented, and confident. The oracle feeds were not. I spent three nights simulating a flash-loan attack against the price input, and I drained a test pool of $150,000 in simulated assets before the developers patched the vector. The exploit did not require understanding the agents' reasoning. It required understanding their inputs.
That experience is why I read interpretability claims with a specific skepticism. The promise of neuron-level interpretability is that we will understand why a model did what it did. But the failures that actually cost money are rarely reasoning failures. They are input failures, oracle failures, assumption failures โ the model was fine, the world it was fed was wrong. Interpretability of the mind does not secure the perimeter. Minting fails when the math breaks trust, and the math that broke was upstream of the model, not inside it. A claim that we can now read the model's neurons answers a question that was not the binding constraint.
The regulatory angle is where this becomes more than academic. The EU AI Act's obligations for general-purpose models began applying in August 2025, with transparency and human-oversight requirements for high-risk systems. Regulators want "understandable information about how a decision was reached." That is a governance requirement written in the language of documentation, not mechanism. There is a vast gap between "provide an explanation a human supervisor can evaluate" and "demonstrate neuron-level interpretability." The headline collapses that gap. It implies the technical substrate for compliance now exists at a granularity regulators never asked for and researchers have not delivered.
This is how trust theater gets built. A regulator reads a headline. A vendor cites the headline. A procurement document references "state-of-the-art interpretability." No one in that chain has run the causal intervention test. No one has checked whether the explanation predicts behavior. The paperwork is coherent and the assurance is hollow. I have seen the financial equivalent โ risk reports that were internally consistent and externally wrong, signed off by people who never opened the model. A flat line is more dangerous than a spike, because a spike triggers investigation and a flat line triggers complacency. This headline is a flat line. It looks like progress. It invites no audit.
The honest accounting of interpretability's track record is uncomfortable for everyone selling it. To date, there is no public case in which mechanistic interpretability successfully prevented a real-world model harm event. Not one. The field has produced genuine scientific advances โ SAEs work, features are real, the decomposition is reproducible โ and zero documented safety interventions where an interpretability finding stopped a deployment. That is not a reason to abandon the research. It is a reason to be precise about what stage it is at. It is pre-deployment science. It is not a compliance product, and it is certainly not a settled result you can put in a headline without a paper behind it.
I want to be fair to the underlying signal, because dismissing it entirely would be as sloppy as the claim itself. Something real is happening. Interpretability is industrializing. The SAE pipeline is no longer an artisanal craft; it is a repeatable process that scales with compute. Feature extraction on production models is a genuine capability that did not exist three years ago. The vocabulary is stabilizing, the benchmarks are being published, and the open releases โ Gemma Scope being the clearest example โ mean outside researchers can now attack claims instead of trusting them. That is progress, and it is measurable.
It is also plausible that Cammarata meant features and the word "neurons" was a shorthand that survived into a headline. In casual technical conversation, "neuron" is often used loosely for "unit of activation," and a researcher speaking off the cuff might use it that way. If that is what happened, the substance is defensible and only the packaging is wrong. But packaging is the entire product when the product is a headline. The qualifier that would have saved the claim โ "features, in some layers, under one metric" โ is exactly the qualifier that was deleted.
The bulls also get one thing right that the skeptics miss. The migration of interpretability from a research curiosity to governance infrastructure is real, and it will create durable demand: audit firms, tooling vendors, safety evaluators, standards bodies. That demand does not depend on this specific claim being true. It depends on the regulatory requirement existing, and it does. So the industry narrative can be directionally correct even when a specific data point is unverifiable. Trust the compiler, verify the intent. The intent here โ make AI legible enough to govern โ is sound. The verification is missing.
What would change my assessment is specific and cheap to produce. A preprint. A model name. A layer range. An evaluation protocol with a held-out set. A causal intervention showing that editing a neuron changes behavior in the predicted direction. A replication by a team with no institutional stake. Any three of those would move this from a mood to a claim. None of them exist in the source material, and the absence is not an oversight โ it is the normal state of a signal that traveled from a short post to a crypto outlet without passing through peer review.
The next interpretability claim that matters will not arrive as a sentence. It will arrive as a benchmark, a replication, and a causal test, and it will be boring to read. That is the tell. Real results in this field look like methodology documents, not breakthroughs. When the next headline says a threshold was crossed, ask for the date, the metric, and the model. If the answer is a name and an affiliation, you have not been given a finding. You have been given a mood with a timestamp problem.
The neuron was never the unit. The paper is. Until someone produces it, the claim stays in the queue with the unaudited contracts โ filed, not trusted.