There is a room I keep returning to in my mind. It is a Tuesday night in Chicago, late 2020, and forty-two people are on a UnityDAO governance call. The same forty-two show up every month — out of three thousand members who technically hold voting power. A proposal sits on the screen, wrapped in the aspirational language governance proposals always use: align the treasury with community values. It passes, 91 percent in favor. Turnout: 4.2 percent. Three wallets, two of them tied to the same early investor, cast 68 percent of the affirmative weight.
The community got its decision. The community was nowhere in it.
I keep coming back to that room because the same choreography is now playing out at a vastly larger scale. In recent weeks, four of the most powerful artificial intelligence companies on earth — OpenAI, Anthropic, Meta, and Microsoft — have publicly asked governments to regulate something they call "self-improving AI." On its face, this reads as responsibility. I want to argue, carefully and specifically, that it reads as something else: the same move that whale made in my conference room, only wearing a lab coat.
To understand why, we have to separate what was actually said from what the coverage implied. The original reporting arrived through a crypto vertical — Crypto Briefing — in a piece barely three hundred words long. No named executives. No venue. No date. No draft legislation. Four claims, three of them paraphrases of one another. That is not a news event. That is a press release dressed as one, circulated to an audience already primed to distrust centralized authority.
Strip away the dressing and the substance is thin. A handful of frontier labs want governments to treat the hypothetical arrival of a recursively self-improving system — an AI that can redesign itself faster than humans can supervise — as a first-order regulatory concern. The concept is not new. It traces to I.J. Good's 1965 speculation about an "intelligence explosion," and it has been revived every few years since, most recently at the Bletchley and Seoul AI safety summits, where the same coalition of labs formed what looked, from the outside, like a chorus.
Here is what the coverage quietly left out. "Self-improving AI" is not a regulated category, an engineering specification, or even a defined term. It is a mood. And when an undefined term becomes the object of a regulatory request, the request is doing political work, not technical work. I have spent twenty-seven years watching institutions use language this way — banks ahead of Basel III, exchanges ahead of MiCA, and now labs ahead of whatever comes next. The pattern is stable enough to name.
I am not an AI researcher by trade. I am a governance architect, and I spent two years building the thing these labs are now describing from the outside: a system where people were supposed to decide, together, what a shared resource should do. What I learned in that work is that the loudest calls for rules almost never come from the people who will be governed by them. They come from the people who will write them.
So let us do the technical work the coverage skipped. The current state of self-improving AI is not a threat at the level of recursive autonomy; it is a threat at the level of automated research assistance. Those are galaxies apart. What exists today is real but bounded: models that generate synthetic training data, that assist hyperparameter search, that help write and debug code, that draft proofs and propose material candidates. What does not exist is a system that proposes a new architectural paradigm from scratch, validates it, and closes the full loop from pretraining to alignment without a human in the circuit.
In my audit work I use a simple ladder. Level zero is AI assisting humans who do research. Level one is AI proposing hypotheses that humans verify. Level two is AI proposing and verifying while humans review outcomes. Level three is the closed autonomous loop — the thing the labs want regulated. Every deployed system I have examined sits between level zero and level one. Regulating level three is not precaution. It is extrapolation sold as foresight.
The reason we stay stuck at level zero is not compute or cleverness. It is the generation-verification gap. A model can propose ten thousand candidate designs in an afternoon; it cannot tell you which three are actually better, because it lacks an internal, reliable criterion for "better." This asymmetry is the actual bottleneck, and it is deeply inconvenient for the existential-risk narrative, because it means recursive acceleration has a hard brake built in. Any claim that self-improvement will outrun supervision has to explain how a system verifies its own improvements — and no one has.
Now watch how the regulatory machinery actually engages. The EU AI Act and the former US executive order both lean on training compute as their measurable trigger — thresholds around ten to the twenty-fifth, or ten to the twenty-sixth, floating-point operations. When you anchor regulation to FLOPs, you are not regulating harm. You are regulating the size of the cluster in the room. And here the economics turn ugly, in the way they always do. Compliance is a fixed cost. For a lab spending a billion dollars on a single training run, a compliance team is a rounding error. For a university lab or an open-weight collective, it is an existential tax.
I watched this exact distortion inside my own world. When UnityDAO introduced a proposal requiring every treasury movement above a modest size to pass a nine-page disclosure review, the intent was transparency. The effect was that only the three members with paid staff could ever move funds at scale. The rules were neutral on paper and fiercely partisan in practice. Compliance burden is never neutral; it is always a regressive tax in disguise.
Apply that lens to the four labs. The coalition is not accidental. When genuine competitors converge on the same regulatory pathway, the pathway usually protects the incumbents. The finance analogy is exact: after 2008, the largest banks supported capital requirements that quietly eliminated hundreds of smaller institutions. Basel III was not a punishment; it was a moat.
Then there is the anomaly nobody in the coverage addressed. Meta's chief AI scientist, Yann LeCun, has been the most public critic of existential-risk framing, arguing it suppresses open development. If Meta executives did join this chorus, one of three things is true: the voices were policy staff speaking more moderately than reported, the remarks were generic endorsements of "reasonable regulation" flattened into a single position, or the reporting is simply wrong. A serious account would resolve this. This one does not, because resolving it would require a second source.
And that, precisely, is the DAO lesson. When I ran those forty-two monthly calls, I learned to distrust unanimity. Genuine communities disagree loudly and constantly. Consensus that appears too clean signals that dissent has been filtered out before anyone took a vote. Four labs singing the same note is not a community. It is a boardroom with a microphone.
Here is the contrarian reading, and it cuts against my own instinct to applaud any call for guardrails. The real danger of the "self-improving AI" frame is not that it over-regulates; it is that it misallocates attention. By pushing the conversation toward a speculative level-three system, the discourse starves the risks that are already here — biased scoring systems, synthetic disinformation, copyright extraction, and the quiet displacement of entry-level labor. This is the same maneuver I saw when DAO treasuries debated "long-term protocol philosophy" while ignoring that four wallets controlled every consequential vote. Grand future threats are a wonderful place to hide from present accountability.
There is a second reason to be wary. The undefined term is not sloppy drafting; it is strategic ambiguity. When a regulator cannot pin down what "self-improvement" means — is it capability self-replication, training-speed acceleration, rising autonomy? — every future rule can expand under its banner. That expansion is written by the compliance teams of the largest players, who are the only parties with the staff to shape it. A vague category is not a safety feature. It is a jurisdiction waiting to be claimed.
I say this as someone who spent the last year negotiating on the other side of that table. When my Values First coalition pushed fifteen smaller DAOs to adopt a shared transparency charter and took a ten-million-dollar grant from an institutional venture arm, we insisted that the money came with our protocols attached, not theirs. We won not because we were loud but because we were specific. We defined terms before we asked for resources. That is the discipline the AI regulatory conversation is missing.
So what should a person who cares about human agency actually watch for? Not the rhetoric. The receipts. Track three things. First, whether any of the four labs couples its call with a binding self-commitment — a paused capability tier, a published training log — rather than a request that others be governed. Second, whether the eventual proposal contains an open-weight exemption, or whether open release is quietly folded into the liability perimeter, which would be the true competitive kill shot. Third, whether the model release cadence over the next six months actually slows, because a lab that asks for caution while shipping faster is not cautious. It is marketing.
The reason I built a manual verification layer over a thousand DAO proposals in 2026 — reading them, one by one, before any vote — was simple: I do not trust automated consensus, and I do not trust self-appointed guardians either. Code without compassion is cold, but code with compassion and no accountability is just a warmer cage. The same holds for AI regulation. A framework that protects people has to be legible to them, checked by them, and revisable when it fails them. Otherwise it is the treasury proposal all over again — technically neutral, socially hollow, and signed by three wallets.
I still think about that Tuesday night in Chicago. We did not fix it with a better pitch. We fixed it by lowering the cost of participation until the quiet members could afford to speak, and by making every proposal small enough that a person could hold it in their head. Compassionate systems are not the ones that promise to care. They are the ones that are built so care is possible.
The next decade of AI will be decided in rooms most of us will never enter. The question worth carrying out of this week is not whether machines will learn to improve themselves. It is whether we will remember how to govern ourselves before we hand the pen to the four companies already writing the rules.