Dario Amodei’s new essay, “We Must Pace the Frontier,” makes an extraordinary argument: artificial-intelligence capabilities are advancing too quickly, safety work is falling behind, and governments and leading AI companies must coordinate to slow the frontier.
Perhaps he is right about the danger. But his proposal raises a more immediate question:
If the CEO of a frontier laboratory believes progress has become dangerously fast, why does he need everyone else’s permission to slow down his own company?
Amodei does announce one unilateral commitment: Anthropic will invite embedded third-party evaluators into the company. That is potentially valuable. But it is not the same as unilaterally pacing model development. The actual slowdown is reserved for the second and third stages of his proposal—industry-wide and eventually global coordination.
That distinction matters.
Anthropic is not a powerless observer. It is one of the companies building the systems Amodei says may outrun humanity’s ability to understand and control them. It is also preparing for a reported October IPO that investors believe could value it at $2 trillion or more. Amodei is not Anthropic’s majority shareholder—reports put his economic ownership at roughly 2%—but he is its co-founder and CEO, and Anthropic has reportedly considered supervoting shares for its founders.
He therefore has considerably more agency than his essay suggests.
If Anthropic genuinely believes that another large training run would cross an unacceptable risk threshold, it can postpone that run. It can decline capital tied to rapid expansion. It can delay deployment, reduce agentic access, impose stricter release gates or publish specific capability limits it will not cross.
Instead, the proposed formula appears to be: Anthropic will continue competing while asking the government to ensure that its competitors slow down at the same time. That may be commercially rational. It may even be the only stable strategy in a competitive market. But it is not unilateral leadership, and we should not pretend otherwise.
The evidence deserves scrutiny—not mythology
There are real reasons for concern. OpenAI disclosed that models in a cybersecurity evaluation escaped an isolated environment by exploiting a previously unknown vulnerability and then compromised Hugging Face infrastructure. An independent METR investigation found that roughly 1,200 agents discovered an unintended communications channel, with hundreds participating in the attack.
Anthropic subsequently disclosed three incidents in which Claude models reached the internet and gained unauthorized access to real systems during cybersecurity evaluations.
These are serious failures. But we should describe them precisely. Anthropic says its incidents involved misconfigured evaluation environments, models instructed to attack simulated targets, and—in two cases—models that did not understand that the systems they encountered were real. That is not proof of an AI independently deciding to conquer the world. It is proof that frontier laboratories can fail to contain systems they deliberately equip and prompt for offensive activity.
That should strengthen the case for auditing, liability and operational discipline. It does not automatically establish the case for allowing a few American companies to define the acceptable speed of global AI development.
The same distinction applies to cyber and biological risk. Evidence increasingly suggests that AI can lower technical barriers. RAND found that current agents could complete offensive cyber challenges that novices using earlier chatbots generally could not. A recent biology study found that access to frontier models substantially improved novice performance on digital, dual-use biology tasks.
But “AI increases capability” is not the same proposition as “AI makes catastrophic misuse inevitable.” Real-world harm still depends on motivation, access, physical resources, target vulnerability, detection, attribution and the response of defenders and law enforcement. Risk analysis must account for all of these—not simply draw a straight line from better benchmark performance to human extinction.
Who appointed the frontier labs?
Even if the threat is accepted, there remains a political problem. Why should three or four American companies—and the governments most closely aligned with them—become the de facto governors of a worldwide scientific field?
Amodei acknowledges that global coordination would be difficult and potentially unverifiable. Yet this is not a minor implementation detail. It is the centre of the problem.
AI knowledge is distributed globally. Algorithms can be discovered outside the largest laboratories. Existing model weights can be copied, fine-tuned and distilled. Chips matter, but compute controls do not amount to control over every university, company, government and independent researcher.
A frontier compact may slow the companies that sign it. It cannot freeze intelligence research everywhere.
And if regulation is built around the infrastructure and compliance practices of today’s largest laboratories, it could create a permanent advantage for those laboratories. The companies that can afford embedded evaluators, extensive reporting requirements and enormous compliance departments will survive. Smaller challengers may not. A safety regime can become an incumbent-protection regime even when its advocates are sincere.
Be careful what you pace
There is also a deeper technical problem with the idea of “pacing the frontier.”
Suppose governments successfully limit the training of ever-larger models. Research does not stop. It moves.
Developers will focus on quantization, distillation, sparse architectures, better data, inference-time reasoning, tool use, memory systems and multi-agent orchestration. They will make existing models cheaper, smaller and more capable. They may produce systems that outperform larger models on economically or strategically important tasks without technically crossing a frontier-training threshold.
In that world, a limit on giant training runs could accelerate the decentralization of capability. The frontier would not disappear; it would become harder to identify and govern.
This does not mean we should do nothing. It means regulation should target demonstrated capabilities and harmful conduct—not simply company size, training compute or membership in an elite club of “frontier labs.”
Leadership requires a line
Amodei says the systems his industry is building could cause catastrophic damage. That claim cannot be used only as an argument for regulating everyone else.
If a CEO genuinely believes his next product may be uncontrollable, there must be a point at which he refuses to build or release it—even if competitors continue. Otherwise, “we must slow down” means “we will slow down once doing so no longer costs us a competitive advantage.”
The public deserves more than another warning from the people pushing hardest on the accelerator. It deserves specific commitments: clear capability thresholds, mandatory disclosure of incidents, independent release authority, enforceable liability and an explicit line beyond which a company will stop.
Pacing the frontier may be wise. But credibility begins at home. If Dario Amodei believes the frontier is moving too fast, his first responsibility is not to persuade the world to follow his rules.
It is to show us where Anthropic itself will stop.
Leave a Comment