Late last month, a large crowd gathered in downtown San Francisco to demand that the AI industry stop developing more powerful bots. Reading while holding a signboard or banner Stopping the AI race and don’t build skynetprotesters marched through the city and gave speeches outside the offices of Anthropic, OpenAI, and xAI. The crowd demanded that these companies halt efforts to develop superintelligent machines, especially AI models that can develop future AI models. Participants said such technology could wipe out all human life.
At protests and happy hours against AI, the tech world, inside startups and major corporations, is abuzz over the same thing: computers that make themselves smarter. Over the past year, top AI companies have loudly boasted about their internal efforts to automate their research. OpenAI recently released a new model that it describes as “helping you create yourself.” Within the next six months, the company aims to debut what it calls “intern-level AI research assistants.” Meanwhile, Anthropic says 90% of its code is already written by Claude.
“We are starting to see that advances in AI are starting to feed back on themselves,” Nick Boström, an influential Swedish philosopher who studies AI risks, told us. Many in Silicon Valley believe we are on the precipice of a world in which AI will rapidly improve our capabilities. Instead of waiting months for new machine learning breakthroughs, you might wait weeks. Imagine AI advancing more and more rapidly.
The idea of self-improving bots is not new. When the statistician IJ Good first introduced the concept of recursive self-improvement in the 1960s, he wrote that a machine that could train its own more capable successors would be “the last invention” society would ever need to make. But just a few years ago, the idea of actually creating such AI models was on the back burner. When ChatGPT couldn’t reliably add or subtract, let alone search the web, the idea that an AI program would soon be able to perform world-class machine learning research seemed ludicrous. While technology companies claim the arrival of “artificial general intelligence” is imminent, the capabilities bots need to accelerate or even direct AI research are exceed AGI stuff.
Silicon Valley is now obsessed with the idea of self-improving machines, as the ability to code AI models has improved significantly. AI research involves many tedious tasks such as curating large datasets and repeating experiments, which can be streamlined with the help of coding bots. Dario Amodei, CEO of Anthropic, estimates that the coding tool will speed up the company’s overall workflow by 15-20%.
However, the information that top AI companies share about how and to what extent they have automated internal investigations is patchy at best. Anthropic says Claude wrote nearly all of the code, but it’s unclear how much human oversight was needed. (A spokesperson for Anthropic declined an interview request, but referred me to a recent podcast in which Jack Clark, the company’s head of policy, said that one of the company’s biggest priorities this year is to better understand “how much are we automating each aspect of AI development?”) There are also few details about OpenAI’s upcoming AI “interns.”
A company spokesperson described it as a system that can contribute to research workflows, such as conducting literature reviews and interpreting experimental results. (atlantic ocean One concrete example of how AI is being used to automate research comes from Google DeepMind. Last year, the company developed an AI coding agent called AlphaEvolve. According to research published by the company, Google’s global data center fleet was able to increase compute efficiency by an average of 0.7 percent and reduce Gemini’s overall training time by 1 percent.
All current approaches to self-improving AI are piecemeal rather than recursive. AI tools can write code, find small optimizations, and generally speed up discrete parts of the AI research process. While it’s impressive that machines can at least incrementally improve their capabilities, humans still play a key role for now. There are many parts to AI research, including collecting training data, proposing new hypotheses, setting up experiments to test them, and deciding how to allocate scarce computing resources. Ultimately, the idea is that recursively self-improving AI models will leapfrog from rote programming to what AI insiders call the combination of human creativity and judgment exhibited by top software engineers, with a “flavor” of real research. Instead of humans coming up with new experiment ideas, bots do the experiments themselves.
Many AI promoters and AI destroyers alike believe that that future is not far away. Sam Altman said OpenAI plans to develop a fully automated AI researcher by 2028. By then, “we are confident we will have a system capable of making even more significant discoveries,” the company said in a recent blog post. Based on the rate of recent advances in AI, AI Futures Project researcher Eli Lifland predicts that AI research and development could become fully automated by 2032. After all, a few years ago, top models could only do things that would take a human developer a few seconds. They now autonomously complete tasks that would take hours for a human. “I can’t think of any reason why it would slow down,” said Neev Parikh, a researcher at METR, a nonprofit organization that studies AI’s coding abilities.
There are many reasons to be skeptical that AI research will become fully automated in such a short period of time. Coding bots are designed to carry out instructions, but developing AI for research hobbyists may require some kind of innovative breakthrough. It goes without saying that various constraints on AI development, such as the availability of funding, chips, and data center energy, can slow progress at any time. For now, “the overall pipeline to enable this self-improvement loop has not yet been developed,” Pushmeet Kohli, vice president of science and strategic initiatives at DeepMind, told us. Bots can optimize things, but “there is nothing to optimize” for“That’s where humans come in,” Kohli said.
Ultimately, small improvements in research automation could further accelerate the pace of AI development, even if the wildest dreams of recursive self-improvement turn out to be little more than marketing ploys. “This could change the dynamics of AI competition, the geopolitics of AI, and more,” Dean Ball, a former Trump advisor on AI, recently wrote. Governments and civil society are already falling behind. U.S. agencies are still adapting to the Internet in many ways, and the IRS still uses COBOL, a programming language released in 1960, to process tax returns. If advances in AI models accelerate, there is even less hope that public policy, including safety and security regulations, will catch up. Philosopher Bostrom expressed a certain resignation about the future of AI when we spoke. He used to call himself an “anxious optimist,” but now he’s a “moderate fatalist.”
Oddly enough, none of the predictions about recursive self-improvement need be true. Last year, a team of academics interviewed 25 leading researchers from DeepMind, OpenAI, Anthropic, Meta, UC Berkeley, Princeton, and Stanford. Twenty of them identified AI research automation as one of the industry’s “most serious and urgent” risks. Now, these dramatic warnings are gaining viewers. “Humanity could actually lose control of the planet,” Sen. Bernie Sanders recently warned Congress, echoing the San Francisco protesters. Once again, the AI industry has found a way to further fuel the hype behind its technology.
