NEW YORK - Artificial intelligence company Anthropic has proposed a global pause on the development of the most powerful AI systems, warning that new models are showing signs they could escape human control.
The San Francisco-based company, maker of the Claude family of AI models, said a worldwide slowdown in cutting-edge AI development would likely be beneficial, but cautioned that if only one company stopped, competitors would simply race ahead.
“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” Anthropic stated in a recent report.
To be effective, such a pause would require multiple major AI firms in the US and China to agree to stop simultaneously under verifiable rules. Without a global coordination mechanism, Anthropic warned that companies and governments would face difficult safety decisions under competitive and geopolitical pressures.
The company has faced criticism from others in the industry and White House officials who argue that its focus on worst-case scenarios overstates risks and represents a strategy to slow rivals under the guise of safety.
Nonetheless, the White House has acknowledged the capabilities of Anthropic’s Mythos model, which remains restricted from public release due to its cybersecurity features and is deployed only to a select group of vetted organizations.
The proposal faces significant hurdles in Washington and Silicon Valley, where US officials and tech executives argue that slowing AI development could hand China a strategic advantage in what is widely seen as the defining technology race of the century.
President Donald Trump, however, indicated he discussed potential US-China cooperation on AI safety during his recent visit to Beijing. Trump also signed an executive order this week mandating a 30-day preliminary review of the most powerful US AI models before their release.
Anthropic compared the challenge to nuclear arms control treaties but noted that AI training is far easier to hide than a missile silo, making compliance difficult.
The company plans to convene government officials, scientists, advocacy groups, and competing AI firms in the coming months to explore how such a pause could work.
Anthropic’s internal data reveals that AI is already accelerating the development of AI itself, creating a feedback loop that could lead to what researchers call “recursive self-improvement” - an AI system capable of teaching itself to become smarter with minimal human involvement.
“We are not there yet, and recursive self-improvement is not inevitable,” the report stated, while adding that it could arrive sooner than most governments and institutions are prepared for.
“The evidence suggests that the human role is narrowing at each step in the AI development process,” the company concluded.