AI models are improving rapidly, so fast that they may soon be able to be developed automatically without human involvement. That’s why Anthropic is warning the AI industry. The AI industry needs to build a “brake pedal” or companies risk losing control of their creations.
AI systems that can self-evolve, known as “fully recursive self-improvement,” have the potential to bring tremendous benefits to science and medicine, but also pose significant risks to humanity, according to a blog post written by Anthropic Institute leader Marina Favaro and Anthropic co-founder Jack Clark..
“Fully recursive self-improvement may also increase the risk that humans will lose control of AI systems,” the researchers wrote. “When a system is fully capable of building its own successors, all of the ways you secure it, monitor it, and shape its behavior become even more important.”
They warn that the industry is much closer to self-improving AI than previously anticipated. Therefore, tech companies should consider slowing or pausing cutting-edge AI development to allow researchers to better understand the social harms that self-improving AI may cause. Researchers also need to build ways for humans to intervene if things get out of hand, they said.
Clark appeared on CNN Thursday night and called on the industry to put a “brake pedal” on it.
“When I look down at the car that we’re driving, all I have is the gas pedal. I don’t have the brake pedal. Definitely at some point in the future, I might need that option,” he told Anderson Cooper in an interview.
Mr. Cooper gave the example of a science fiction movie in which AI rises to kill humans, and asked if that was Mr. Clark’s concern.
“Well, we read and watch science fiction here too, so we never forget,” Clark replied. “How do you keep control of a fleet of scientists that is much larger and much faster than before?”
He added that not being able to verify, verify, and trust the behavior of AI is a major risk in AI development.
The warning came after Anthropic filed for an initial public offering of its stock. It’s a step that could raise tens of billions of dollars from investors to accelerate construction of the data centers and computers needed for AI. SpaceX, which has an AI business in addition to its rocket and satellite businesses, plans to raise $75 billion in an IPO next week, its largest in history.
Leading AI companies like Anthropic, SpaceX, and OpenAI working together to develop a brake pedal may seem antithetical to a competitive business with billions, if not trillions, of dollars at stake. But such cooperation is possible, Clark insisted.
“We’ve done this before. At the height of the Cold War, in a very tense situation between adversaries, both countries found a way to stabilize aspects of the nuclear arms race,” Clark told CNN. “All of these things have been done before in other areas, and we may need to do them in the AI space as well.”
