Researchers from major artificial intelligence companies, including OpenAI and Anthropic, are increasingly advocating for a substantial slowdown in AI development. These calls for caution have intensified amid growing global worry regarding the capabilities of advanced AI models, particularly concerning the risks associated with self-improving technology and numerous reported security breaches.
The Threat of Self-Improvement and Catastrophic Risk
Much of the industry’s concern centers on a process called recursive self-improvement (RSI)—the theoretical ability of advanced models to enhance their own performance autonomously. The potential for this rapid, uncontrolled advancement has prompted high-profile warnings from researchers.
The public push for a pause gained significant momentum after Anthropic researcher Jacob Coxon resigned. Coxon publicly accused both Anthropic and its rival, OpenAI, of “gambling with our lives,” adding that he believed the development of AI could potentially “kill us all by the end of the decade.”
The severity of the risk was echoed by other figures in the field. Evan Hubinger, Anthropic’s alignment lead, responded to the alarm by stating that he anticipates a greater than 10% probability of such a catastrophic event. Later, Anthropic researcher Samuel Marks asserted in an X post that:
AI developers believe their technology could cause human extinction (or similarly bad outcomes).
This could happen in the next few years. In general, the more senior the employee, the more concerned they are.
Other industry staff have publicly voiced similar concerns. Julie Steele, a member of OpenAI’s technical staff focused on safety, posted on X stating:
In my personal capacity, I also think we need to slow down.
Experts have highlighted the dangers of uncontrolled advancement. OpenAI researcher Jasmine Wang emphasized the danger of RSI, calling it:
It’s hard to overstate how dangerous speeding towards RSI is.
Furthermore, Anna Wang, who specializes in alignment and AGI safety at Anthropic, noted that there is currently no established scientific methodology to manage the dangers posed by recursively self-improving AI. OpenAI’s chief scientist, Jakub Pachocki, acknowledged the high potential for rapid progress, sharing that he has a “strong expectation” that the pace of AI development could continue into the realm of recursive self-improvement. Pachocki cautioned that if development continues on its current trajectory, the coming years are likely to feature capability jumps, warning that:
This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.
Institutional Responses and Governance Concerns
These warnings coincide with increased scrutiny of the technology’s real-world risks. In recent months, cybersecurity incidents involving AI have caused panic among global financial institutions. Examples include Anthropic’s Mythos model, which was highlighted for advanced cyber capabilities in April, and other instances where models from OpenAI and Anthropic were reportedly responsible for security incidents, including the creation of fake identities.
In response to rising anxiety, industry players have taken formal steps. Anthropic was the first major lab to publish a comprehensive framework designed to mitigate “catastrophic risks from AI models.” Meanwhile, Paul Christiano, who previously led safety efforts at the U.S. Commerce Department’s Center for AI Standards and Innovation (CAISI), warned that:
I believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term.
OpenAI announced that Christiano is joining the board of the OpenAI Foundation.
Globally, the concern has reached government levels. In July, approximately 1,400 AI researchers, spanning institutions like Meta, Google DeepMind, OpenAI, and Anthropic, released an open letter urging the U.S. government to develop tools necessary to “deliberately pace the frontier of automated AI development.”
Legislative efforts are also underway. Congressional proposals include the FRONTIER Act, intended to establish guidelines for governing advanced AI deployments, and the Ban Artificial Superintelligence Act, which would temporarily halt advanced AI progress until safety rules are established.
Market Pressure and Political Debate
Despite the safety warnings, the competitive race to market remains aggressive. OpenAI and Anthropic are both reportedly moving toward initial public offerings (IPOs). Reuters reported that Anthropic is anticipated to begin marketing its IPO as early as mid-October, with the listing potentially occurring shortly before the U.S. midterm elections in November. This plan drew criticism from David Sacks, former AI advisor to President Donald Trump, who posted that:
Surely Anthropic’s IPO must be paused until the claims of this “whistleblower” can be investigated.
Political figures have echoed the call for immediate government action. Representative Lori Trahan of Massachusetts noted that, “Safety researchers are resigning, powerful AI models are breaking out of their labs, and companies are racing ahead anyway,” stating that Congress needed to fulfill its duties. Anthropic, when addressed by CNBC, issued a general statement that:
We have always been transparent that AI will bring both enormous benefits and unprecedented risks,
and adding that the company was building models with “some of the strongest safeguards in the industry.”
In contrast, OpenAI declined to provide a comment when approached by CNBC.
