Artificial intelligence company Anthropic on Thursday proposed a global pause on building the most powerful AI systems as modern models begin to show signs of escaping human control.
The San Francisco-based company, which makes the Claude family of AI models, said in a report that a global slowdown in cutting-edge AI development is “probably a good thing,” but warned that if just one company stands still, rivals will simply jump ahead.
“We believe the world is better off with the option of slowing or temporarily halting frontier AI development so that social structure and coordination research can catch up with technological advances,” the group said.
An actual shutdown would mean multiple large AI companies in multiple countries (particularly the U.S. and China) agreeing to shut down at the same time under rules that everyone can actually verify, Anthropic said.
“Without a global coordination mechanism, businesses and governments will have to make difficult safety decisions in the face of competitive and geopolitical pressures,” the report said.
The company has faced pushback from industry insiders and White House officials, who argue that focusing on worst-case scenarios exaggerates risks and amounts to a strategy to slow down competitors using safety concerns as cover.
Still, the White House has acknowledged the power of the company’s Mythos model, which is closed to the public due to its cybersecurity features and is currently deployed only in a small number of vetted organizations.
The proposal will face a tough battle in Washington and Silicon Valley. U.S. officials and technology executives have repeatedly argued that slowing AI development risks giving China a decisive strategic advantage in what many see as the defining technology race of this century.
However, US President Donald Trump said during his recent visit to Beijing that he discussed the possibility of cooperating with China on AI safety issues.
President Trump also signed an executive order this week allowing the government to conduct a 30-day preliminary review of the most powerful American AI model before announcing it.
“Narrowing of human roles”
Antropic likened the issue to nuclear arms control treaties, but said AI training would be much easier to hide than missile silos and even harder to address because the temptation to keep quiet is so great.
The company said it plans to convene government officials, scientists, advocacy groups and competing AI companies in the coming months to figure out how such a system would work.
Anthropic said this call for adjustment is also accompanied by internal data showing that AI is already dramatically speeding up its own development.
This acceleration could create a feedback loop that could eventually lead to what researchers call “recursive self-improvement,” Antropic warns.
This is the idea of AI systems that can essentially learn on their own and become smarter, with little to no human help.
“We are not there yet, and reflexive self-improvement is not inevitable,” the report said, but added that it could get there sooner than most governments and institutions are ready.
“There is evidence that the role of humans is narrowing at each stage of the AI development process,” the company said.
