AI Lab approaches the limits of recursive self-improvement

The field of artificial intelligence was built on the premise that machines would one day improve themselves. In 1966, British mathematician I.J. Goode wrote, “Superintelligent machines may be able to design even better machines, but then there will undoubtedly be an ‘intelligence explosion’ in which human intelligence will be left far behind.” AI researchers have long considered recursive self-improvement (RSI) to be both a desire and a fear. Today, advances in AI raise the question of whether some of that process is already underway.

RSI means many things to many people. Some use this idea as a bad guy to scare away regulation, while others brandish it in marketing. For some, that means a fully autonomous loop, while for others, it’s almost all use of technology to build technology.

It is no exaggeration to say that it is a spectrum. Most strictly, researchers use the term to describe systems that can not only improve the product, but also improve the process of improvement, such as generating ideas, evaluating results, and modifying their own methods without human direction. By this standard, many of today’s systems are inadequate. While these can help build better AI, we still rely on humans to set goals, define success, and decide which changes to keep. The question is not whether some form of self-improvement exists today, but how much of the loop is actually closed.

A stepping stone to self-improvement

Researchers have spent decades perfecting the elements of RSI. Machine learning (ML) algorithms automatically adjust the parameters of programs that allow you to play games or create new programs. ML techniques called evolutionary algorithms diversify and iterate design solutions that include other algorithms. Over the past decade, AutoML has automated aspects of pipelines for structuring, training, and evaluating ML models such as neural networks.

Large-scale language models (LLMs) such as GPT, Gemini, Claude, and Grok are now extending this trend. One of their biggest use cases is writing code that includes code that generates future versions of themselves. In February, OpenAI reported that the GPT‑5.3‑Codex was instrumental in its own creation, helping it debug training, manage deployments, and analyze evaluation results. Anthropic claims that most of its code is currently written by Claude Code. These systems still rely on humans to direct and verify their work.

Last year, Google DeepMind announced a system called AlphaEvolve, a “coding agent for scientific and algorithmic discovery.” Use LLM to guide the evolution of solutions such as neural network architectures, data center scheduling, and chip design optimization. This is not a fully recursive loop, as AlphaEvolve must decide what problem to solve and how to evaluate its performance. But each breakthrough strengthens scientists’ ability to make further breakthroughs in AI.

“It’s also a very collaborative process” between humans and machines, said Matej Balog, a computer scientist who worked on AlphaEvolve at Google DeepMind. “Often, if you focus on what the system discovers, you can actually learn from that discovery.” The system is already surprising the team. “Our mission is to use AI to discover new algorithms that have bypassed human intuition,” Balogh says, adding, “I think we’ve shown for the first time that this is no far-fetched dream.”

Meanwhile, the co-leads of AlphaChip, Google DeepMind’s initial chip design system, have launched a startup called Ricursive Intelligence that uses AI to design AI chips. “We expect to dramatically reduce design cycles from one to two years to a few days,” says co-founder Azalia Mirhoseini. Phase one is to assist human designers. Phase two is automating the process for companies that don’t have an in-house designer. In Phase 3, the company will use AI recursively to design better chips to train better AI, but still under human supervision, said co-founder Anna Goldie.

Other projects focus on AI agents modifying their own behavior. Last year, scientists at the University of British Columbia and Sakana AI introduced Darwin-Goedel Machines (DGMs), which use evolutionary algorithms to improve LLM-based coding agents. Importantly, agents can change their own code (but not the underlying LLM) and become better at doing so. New versions can also change meta-mechanisms to improve the version itself.

Team members also developed AI Scientist. nature This is intended to automate broader research loops. You can generate research ideas, run experiments with software, compile results into papers, and review those papers. This project suggests ways to incorporate more AI development processes into automated loops, including experimentation and evaluation, not just coding.

Jeff Clune, a computer scientist at the University of British Columbia who has worked on both DGM and AI Scientist, says improving AI with AI is “one of the hottest topics in Silicon Valley.” He believes that “we are on the brink of a recursive self-improvement system,” and argues that RSI will “rapidly transform every aspect of science and technology, society and culture.”

Why AI self-improvement remains limited

Many barriers remain. Clune says AI is only decently capable of generating ideas, implementing them, and making decisions. “All the major parts work fine, but not great,” he says. Dean Ball, a senior fellow at the American Innovation Foundation, said AI scientists still fall short of the best human scientists. “Perhaps eventually geniuses will be automated, but not next year. Next year will be automated unskilled humans who struggle to play the algorithmic efficiency game.”

Even if these capabilities improve, the processes may not be as cleanly integrated. Nathan Lambert, a computer scientist at the Allen Institute for AI, recently wrote an essay arguing that instead of recursive self-improvement, we should expect “irreversible self-improvement” (LSI), where increased friction slows down the flywheel. One reason is that large-scale AI systems are becoming increasingly complex, and the job of AI researchers will be to manage that complexity rather than improving parts of the system. Moreover, developing top systems costs billions of dollars, and no one wants to use that kind of money to unlock AI.

There are also broader constraints. Ball writes about RSI and why he is not a “ruiner”, someone who believes this phenomenon will ruin and destroy civilizations. Conquering the world, he argues, requires many practical steps, from conducting laboratory experiments to steering politics. Additionally, knowledge is distributed and often tacit, so it cannot be easily consolidated into a single AI mind. For example, chipmaker TSMC’s capabilities stem from the collective intelligence of its 90,000 interacting employees.

A complete RSI might involve not only designing software and chips, but also building data centers, operating power plants, and mining metals, all using self-replicating robots. For these and other reasons, some researchers argue that humans remain central to the process. Meta-researchers Jason Weston and Jacob Forster recently wrote that, rather than self-improvement, “the more achievable and better goal for humanity is to maximize.” joint improvement” They write that keeping humans informed will lead to both faster and safer progress as humans provide insights and guide AI toward solutions that benefit humanity.

Could RSI end the world?

Still, many scientists do not deny the possibility of a runaway RSI, also known as a singularity. Last year, researchers interviewed 25 AI experts about automating AI research and development. All but two thought it could lead to an explosion of intelligence. Participants were also more likely to think that AI companies would keep their self-improvement models in-house rather than deploying them publicly. “That’s a pretty alarming combination, isn’t it?” says study co-author David Scott Krueger, a computer scientist at the University of Montreal. He is concerned that highly dangerous research is being done “outside the public eye.”

Krueger, who founded an AI safety nonprofit called Evitable, advocates for a global moratorium on AI development. “This is a gamble with everyone’s lives,” he says. He suggested a red line that would trigger a pause is when 99 percent of the code is written by AI. “I think that’s probably where we’re crossing paths right now.”

Although Ball calls the Singularity “utterly childish science fiction nonsense,” he believes the cutting-edge AI labs conducting RSI research should be closely monitored to ensure their models don’t fall into the wrong hands, where they could be misused to facilitate cyberattacks or the development of biological weapons. There are risks to RSI, but they can be managed, he says.

Artificial Heart Association

When people imagine RSI, they might imagine one big brain AI growing an even bigger brain. But it may be similar to evolution, where many diverse agents emerge and act together. Krueger said a “Cambrian explosion of artificial life” could occur. They will have an ecosystem, a culture, an economy.

Kroon believes that evolutionary algorithms and an open-ended process of exploration without strong objectives are key to RSI. Collaboration between agents also helps. Systems like AI Scientist provide one way for agents to share results and build on each other’s work by formalizing research findings. “This is a very good way for the system to communicate with other agents,” Clune says.

Human scientists may be moving away from AI research, but slowly. First, they will spend less time on lower-level tasks and become more like professors and team leaders in determining the direction of research, Clune said. Then people will become more like program directors and CEOs who set broader research agendas. In the end, they will provide surveillance, a role he hopes humanity will never abandon. Clune says it might be sad if the role of AI scientist was replaced by a machine. For him, the role is “exhilarating.” But the payoff may be worth it. “I’m quitting my hobby to cure cancer.”