The people asking AI to slow down are the people who get rich if it doesn’t.
That should stop you, and it is worth sitting with before you read another word.
Two days ago Dario Amodei published an essay called “We Must Pace the Frontier”, arguing the industry should deliberately slow how fast models gain capability. Within hours Sam Altman committed to match the first step. Elon Musk wrote three words: “Dario is right.” The market discounted AI-exposed valuations.
They said it anyway.
Costly signals are the only kind worth reading
I teach a version of this to executives at Berkeley and it applies cleanly here. When someone tells you something that costs them nothing, you have learned about their marketing. When someone tells you something that costs them money, you have learned about their beliefs.
These men run the labs. Their valuations are built on the assumption that capability keeps compounding on schedule. Arguing publicly for a slower frontier is an argument against their own quarter, and the market charged them for it inside the same week.
That is a costly signal. It does not prove they are right. It does prove they mean it.
Now notice who is not asking.
Not the regulators. The President spent the week emphasizing America’s lead over China. Not the legislatures. Not any government with subpoena power or inspectors. The alarm is coming from inside the building, and the building holds the only instruments.
That asymmetry is the entire story. The people with the measurements are frightened. The people with the authority are not. When the only parties who can see the dial are the only parties asking to slow down, the reasonable inference is not that they have gone soft. It is that they are reading something the rest of us cannot see.
So what are they reading?
They have told us, publicly and in unusual unison. Amodei, Hassabis and Altman jointly called for laws against AI-assisted bioweapons, warning that these systems now outperform PhD-level virologists on many biotechnology questions. Ask people in this industry what actually keeps them up and the answers converge with unsettling consistency on one scenario: a pathogen, engineered, moving too quietly and too fast to contain.
Anthropic then published what it had actually caught.
Roughly thirty-five separate efforts in a single thirty-day window, attempting to route around safety controls. Bird flu. Orthopoxviruses. Novel venom toxins. In one case from May, a researcher affiliated with a military institute asking for help drafting a grant application for gain-of-function work on chikungunya.
And then the line that nobody should skim past:
Their newer models can no longer be assumed to sit below the threshold for meaningful bioweapons assistance.
That is the first time a frontier lab has said that out loud about its own system. Not a think tank. Not a critic. The lab, about its own model, in public.
My question is simple, and I have not found anyone with a satisfying answer to it.
That is what the monitoring caught. What about what it did not?
Thirty-five interventions in thirty days is not a reassuring number. It is a sampling rate. It tells you the detector is working. It tells you nothing about the denominator.
The fiction was optimistic
We have had the parables for two decades and we have learned almost nothing from any of them.
The Animatrix imagined how the handoff happens, and looking at it now it reads as generous. It assumed the machines would want us alive, farmed and dreaming. It never considered that the danger might be engineered by us first, with the machine as a very capable lab assistant. It never considered that a system could hand a motivated human a capability no state actor had granted them.
Optimization needs no malice. It never did. That was the whole point of the paperclip argument and we filed it under science fiction because it arrived wearing a costume.
And Tim Urban asked the question in 2015 that still has no answer. Not what happens in the decade after we reach the threshold. What happens in the sixty seconds after.
Nobody has answered that. Not in eleven years. We have simply gotten better at not being asked.
If you have read this far and someone specific came to mind, a founder, an investor, a legislator, someone who can actually move a policy lever, forward this to them now rather than at the end. That is the only distribution that matters for a piece like this.
Which brings me to my neighbors
There are more than 240 billionaires living in California, holding over two trillion dollars between them, according to work out of Berkeley by Saez and Zucman. Worldwide, more than 3,300, holding north of twenty trillion.
You do not just fund this race. You hold the policy. Not metaphorically. The compute gets built where you finance it, the labs incorporate where your counsel advises, and the bills that would constrain any of it live or die on rooms you are in and I am not.
And California is where this gets decided first. This state writes rules the world copies. It happened with vehicle emissions. It happened with privacy. It will happen here. Act local, and it travels global on its own.
So I want to talk about the insurance you have been buying.
The islands. The bunkers. The compounds with the blast doors and the five-year food stores and the hydroponics. I understand the instinct. I want to point out that it is two doors, and both of them close.
Door one: you seal yourself in and it works. You survive. You spend the remainder of your life inside a structure with no community to belong to, no one to build with, nothing to be useful for. That is not survival in any sense a human being would recognize. It is exile, at a premium, chosen in advance.
Door two: you step outside. And the bird carrying the new flu lands on your roof, because a pathogen does not respect a property line and air does not check the guest list.
A blast door filters a mob. It does not filter an aerosol. The threat these men are describing is the one category of risk your money cannot build a wall against, because the wall has to let you breathe.
You are not above this. You are inside it, with better furniture.
The button
We are currently debating whether there should be an off switch. What it would look like, who would hold it, whether it is even coherent to imagine one given China and open weights and the fact that nobody can unilaterally stop anything.
Those are real objections and I do not have a clean answer to them. I am a practitioner, not a policy expert, and I would rather say that plainly than pretend otherwise.
But I know this much. The debate about whether the button exists is only worth having while it still does.
Use the stop button, the one we still think exists, before it disappears.
Anand Vallamsetla writes about responsible AI, agentic systems, and what it takes to keep building with these tools honestly. Previously engineering leadership at Google, Entrepreneur in Residence at AI Fund, and four and a half years teaching AI strategy and responsible deployment to Fortune 500 executives at UC Berkeley Executive Education.
Everything cited here, in one place:
Dario Amodei, “We Must Pace the Frontier”
Anthropic, “Countering misuse of AI: September 2026” (the threat report)
AI executives’ joint warning on bioweapons (Semafor) and the private fear behind it (Axios)
Saez & Zucman, California Billionaires (UC Berkeley) and the Forbes 2026 Billionaires List
Tim Urban, “The AI Revolution: The Road to Superintelligence” (Wait But Why, 2015)
The Animatrix and the paperclip argument, for anyone who has not met them