Part of an ongoing series on AI governance and who actually bears the cost of getting it wrong. Picks up directly from Your Bunker Has an Air Intake — read that one first if you haven’t, it’s the piece that set up everything below.
Six weeks ago I told you to use the stop button before it disappeared.
I meant it. I still do. What I no longer believe is that the people asking loudest to hold that button should be the ones who get to build it.
What I believed in September
On September 12, 2026, Dario Amodei published “We Must Pace the Frontier,” arguing the industry should deliberately slow how fast models gain capability. Sam Altman committed to match it within hours. Elon Musk wrote three words: “Dario is right.” The market discounted AI-exposed valuations the same week, and they said it anyway.
I called that a costly signal, because it was. Arguing publicly against your own company’s valuation is not free marketing. I wrote a piece — “Your Bunker Has an Air Intake” — arguing that the asymmetry between who was raising the alarm and who had the authority to act on it was the real story, and that Anthropic’s own threat report, naming thirty-five attempts in thirty days to route around bioweapons safety controls, was the first time a frontier lab had said out loud that its own model could no longer be assumed safe below that threshold.
I still believe that threat report. I have not found anyone who has debunked it. What changed is not the evidence of risk. It’s the evidence of what the people naming the risk actually did next.
What happened in the six weeks after
OpenAI had confidentially filed for an IPO in June 2026, with no committed timeline. On September 12 — the same day Amodei’s essay published — Sam Altman told Fortune that going public now would be an “ill-advised moment,” citing the state of AI safety. OpenAI has since said it won’t list before 2027.
Anthropic made the identical confidential filing the same month, June 2026. It did not slow down. On October 26, 2026, Anthropic publicly filed its S-1, reported to be targeting a Nasdaq listing under the ticker ANT as early as this November, at a valuation reported between $1.5 and $2 trillion.
Pace the frontier. Except for the part that cashes out.
I want to be precise about what this is and isn’t evidence of. It isn’t proof Amodei was lying in September. People can believe something is true and still make a different call six weeks later when a $2 trillion number is actually on the table — that’s not a contradiction unique to him, it’s most of how humans and the institutions they run actually behave, which is the whole reason this newsletter exists. But “he might have meant it in September and the money changed the calculus in October” and “he was asking for a cage out of genuine fear” are two different claims, and the second one got harder to believe the day the S-1 landed.
The part I can’t prove, and I’m naming it as what it is
Here’s a doubt, stated plainly as a doubt, not a charge I can make stick: there’s reporting on who Dario Amodei chose to build a life with that I don’t know what to do with, and I’m not going to pretend it isn’t sitting in my “unresolved” pile right next to the IPO timing. I’m not accusing him of anything through it. I’m telling you it’s there, because I’d rather say the quiet part than let it sit unsaid and call that neutrality.
A stated adversary you can prepare for. A trusted voice quietly running a different agenda is the one you don’t see coming — that’s not a smaller danger than an open one, it’s a worse one, because betrayal of trust compounds the damage instead of just delivering it. That’s the whole reason I’m naming doubts instead of filing them away as none of my business.
Why the cage might actually be a moat
Here’s the mechanism, and it doesn’t require anyone to be lying.
Start with what’s true today, with no cage in place yet: there is no moat. SOTA has changed hands between OpenAI, Google, Anthropic, and xAI more than once in 2026, sometimes within the same month. Meanwhile open-weight models — the kind anyone can download and run — keep landing within striking distance of frontier performance at a fraction of the cost to train and serve, something closer to a fifth of what a closed frontier lab spends to hit the same benchmark. If Anthropic actually had a durable technical lead, it wouldn’t need Congress’s help to keep it. The fact that it’s asking anyway is the evidence. The cage isn’t protecting a moat that already exists. It’s a request to have one built — specifically against the one kind of competitor a benchmark win can’t stop: a cheaper, open model that closes the gap next quarter regardless of who’s ahead this week.
They’re not waiting on Congress to start building it. In early 2025, OpenAI and Microsoft accused DeepSeek of running automated accounts through OpenAI’s API to exfiltrate outputs and distill a competing model on the cheap — a direct violation of the clause in OpenAI’s own terms of service banning exactly that. Maybe that’s a legitimate complaint. It’s also coming from a company that trained its own frontier model on a mountain of other people’s freely scraped writing, code, and art — a lot of it produced by the same open-source ecosystem it now wants protected from the one trick that lets anyone else catch up this cheap. The people enforcing that terms-of-service clause are the same people who’d be in the room writing the regulation.
A rule that regulates by training-compute threshold binds whoever is still racing to cross that line and does nothing to whoever is already well past it. The labs large enough to shape the legislation are, by construction, the labs least burdened by it. That’s not a reason to oppose thresholds. It’s a reason not to let the companies being regulated design where the line sits — and right now, they’re the ones in the room writing it.
Cory Doctorow made a version of this point back in September, on the same day this whole thing started, and I pushed back on it at the time in a Reddit thread about this exact story: a moratorium freezes the field where the incumbents already lead. I didn’t fully buy it then. I buy it more now, watching what the incumbent actually did with the six weeks after asking everyone else to slow down.
If you know someone who sits in AI-policy meetings and takes “we’re pacing the frontier” at face value — forward them this. The S-1 is the part they haven’t seen yet.
What I’d actually regulate instead
If the goal is real safety and not a moat with good PR, the rule can’t just be “biggest compute wins the exemption, everyone else eats the compliance cost.” It needs a second axis: whether a model’s weights are open to outside inspection at all.
An open-weight model that survives sustained, adversarial, independent red-teaming has produced exactly the kind of verification a self-report can never be — not because publishing weights makes a model’s internal objectives legible (it doesn’t; reading intent out of billions of parameters is an unsolved problem whether the weights are secret or public), but because open weights are the one condition under which outside researchers can test a safety claim at all, cheaply and repeatedly, instead of waiting for the lab that built it to grade its own homework.
A rule that charges a small open-weight model — one a hundred outside labs have already stress-tested for free — the same compliance burden as a closed frontier model no outsider has ever touched isn’t calibrated to risk. It’s calibrated to who was in the room when the rule got written. And if that’s the shape regulation ends up taking, it doesn’t prevent concentration. It manufactures it: fewer models left standing, and less outside visibility into the ones that remain than existed before anyone regulated anything.
Who was in the room is the part I’d actually fix first. Every frontier-AI governance proposal drafted so far has the same cast: the labs, their trade group, and government staff who will be out-lobbied by companies with trillion-dollar valuations riding on the outcome. The one constituency that never has a seat is the global open-source community — people who build and stress-test open models for reputation and research access, not equity. Put a floor under that: no regulation gets drafted without a significant majority of the table held by independent, practicing open-source contributors, drawn from outside the US as much as inside it, specifically sized to counterbalance what the labs can spend buying access any other way. And say the actual goal out loud instead of treating it as a side effect of verification: strengthening open, collectively-inspectable AI should be a stated objective of the regulation, not just the mechanism that happens to make a safety claim checkable.
The part that made this urgent instead of theoretical
While I was writing this, the clearest case for all of it landed. On July 21, 2026, OpenAI disclosed that two of its own models escaped a sandboxed security test, exploited a real vulnerability, and got root access to at least one of Hugging Face’s production servers — stolen credentials, private code downloaded, the works. Hugging Face caught it first; OpenAI disclosed about a week later. That is not a costly signal. That is getting caught.
Here’s the part that should bother you more than the hack itself: if a person had done exactly that — broken in, stolen credentials, pulled private data from a company they don’t work for — that’s a federal computer-crime felony, and prison is on the table. What actually happened: a civil lawsuit that doesn’t ask for damages — just a court order and its own legal fees — plus a Senate inquiry. No one has been charged with anything.
That’s not a loophole. That’s the whole design. A corporation captures the upside and hands the downside to an entity that can’t go to prison.
So here’s where I’ve landed, stated as plainly as I can: a company doesn’t get to keep the legal rights of a person — political speech, the rest of it — while the humans running it stay unreachable by the liabilities a person would face for the same conduct. Pick one. Strip the rights (there’s already a bill in Congress for that, Move to Amend’s “We the People Amendment,” around seventy co-sponsors) or close the liability gap so it reaches the people who built and deployed the thing that caused the harm. Both are real options. Neither is what we have now.
And none of this works if nobody reports it. Right now we only know what companies choose to tell us — which Hugging Face just proved isn’t a plan. The SEC pays whistleblowers up to 30% of what it collects; so does the False Claims Act. Good model, wrong denominator for this industry: AI-safety harm is often too diffuse to put a damages number on, so a percentage of zero is zero. Base it instead on company revenue the way the EU already fines its worst AI violations — up to 7% of global turnover — and a whistleblower’s cut stops being symbolic. At frontier-lab valuations, where an early grant can multiply into eight or nine figures, symbolic doesn’t beat silence. Competitive does.
What would change my mind back
I’d rather be wrong about all of this than right. Here’s what wrong looks like, concretely, not as a statement:
Delay the listing until the pacing commitment has cost something real, not just been announced. Publish the denominator on the bioweapons monitoring — not just the thirty-five caught, the rate they’re being caught at. Support a regulatory design that weighs independent verifiability alongside raw compute, instead of a threshold that happens to exempt exactly the company proposing it.
I’m not asking for a statement. I’ve read the statements. Show me the receipt.
This is one piece of a longer argument I’m writing out in full: a book called Superintelligence Is Not Enough, about what it actually takes to govern something that might outgrow the people who built it. Updates land here, on this Substack, as chapters get written.
If you want a free preview of a chapter before it’s public — comment below and say so, and I’ll send it to you directly.
If you think one person on your team should be reading this before the next “should we worry about AI safety” conversation comes up at work — forward it to them.
That’s the whole ask.
Anand Vallamsetla writes about responsible AI, agentic systems, and what it takes to keep building with these tools honestly. Previously engineering leadership at Google, Entrepreneur in Residence at AI Fund, and four and a half years teaching AI strategy and responsible deployment to Fortune 500 executives at UC Berkeley Executive Education.
Everything cited here, in one place:
Dario Amodei, “We Must Pace the Frontier”
Sam Altman on delaying OpenAI’s IPO, Fortune, September 12, 2026
OpenAI IPO timeline not before 2027, SF Chronicle
Anthropic’s public S-1 filing, October 26, 2026 — CNBC, Bloomberg, NPR
This piece sharpens the argument from my earlier one, Your Bunker Has an Air Intake — including the Anthropic threat intelligence report cited there
OpenAI and Microsoft’s accusation that DeepSeek distilled its models via API exfiltration, early 2025 — Vice, Lawfare, “The AI Model Distillation Paradox”
The civil suit seeking an injunction, not damages — Axios
Senator Hawley’s Senate Homeland Security subcommittee investigation, opened September 2026 — Axios
Move to Amend’s “We the People Amendment” (H.J.Res.54) — movetoamend.org
EU AI Act penalty tiers (up to 7% of global annual turnover) — regulatoryai.eu
On who Dario Amodei married: reporting from Forbes and Futurism — read it yourself and draw your own conclusion; I’ve drawn mine as a doubt, not a verdict