OpenAI has set itself the goal of creating an AI tool that can research its own improvements. The company is now preparing for the risks involved.
The dramatic advances in OpenAI and Anthropic’s coding tools over the past six months have brought the potential for AI systems to achieve so-called “recursive self-improvement” to the forefront among AI leaders. Google DeepMind CEO Demis Hassabis also said this week that humanity is currently at the “foot of the singularity,” the moment when AI begins to improve itself and surpass human intelligence.
OpenAI, which is aiming to go public this year, recently posted a job posting seeking safety researchers to work on what happens when AI trains better versions of itself.
The number of openings for OpenAI’s Preparedness safety team has increased this month, according to the job posting site. The company is offering a whopping compensation package of $295,000 to $445,000 and is looking for a “strong technical executive who supports recursive self-improvement readiness.”
“This research relies on reasoning about problems that may exist in the future but may not exist today,” the listing says. “Therefore, it is especially important that the person filling this role is sensible and strategic.”
OpenAI and Anthropic did not respond to requests for comment from Business Insider.
Top AI labs compete to build self-training models
Models from labs like OpenAI and Anthropic are improving at a dizzying rate, as evidenced by the complexity of the problems they can handle. Researchers at METR, an institute that studies model capabilities, wrote in March that the length of tasks that frontier AI models can complete is doubling approximately every seven months. This means that these models are increasingly able to perform tasks that would take a long time for humans.
What that means, METR writes, is that AI agents will be able to handle “the vast majority” of software tasks that would take days or weeks for human programmers to complete.
OpenAI is actively pursuing this vision, and selling Codex coding tools to enterprises is a significant revenue driver. The company also wants to automate its own research work. CEO Sam Altman said in October that the company had set a goal of running “automated AI research interns” on hundreds of thousands of chips by September of this year and having “a true automated AI researcher in operation by March 2028.”
“We may not fully achieve this goal,” Altman wrote of X, “but given the extraordinary potential impact, we believe it is in the public interest to be transparent about this.”
In April, Anthropic published research on using AI models to monitor more powerful AI models, with promising but limited results. Jack Clark, the company’s co-founder and head of policy, wrote in May that he believes there is about a 60% chance that AI research and development will be conducted without human involvement by the end of 2028.
OpenAI is preparing for self-improving AI
If AI models could train themselves, it would be possible to imagine sci-fi dystopias where their capabilities would skyrocket, escaping containment, and wreaking widespread havoc. This is fear for the AI safety movement. METR CEO Elizabeth Burns wrote on Friday that in her opinion, “a ‘rational’ civilization would clearly use AI to proceed with things more slowly and deliberately.”
OpenAI’s job postings hint at how the company is preparing for a world where AI models can rapidly improve themselves.
Researchers can now focus on protecting OpenAI’s models from data poisoning (attempts to corrupt AI models through the datasets used for training). Employees may also create tools to interpret model inferences and experiment with models to understand safety and hazards.
The post also notes that researchers could “track progress towards automating technical staff,” including measuring the use of AI coding tools.
OpenAI’s Preparedness team is tasked with preventing serious harm caused by AI. The company’s jobs page includes other roles on its team, including an automated red team for testing OpenAI cybersecurity, biological and chemical risks, and agent-based AI threats.
“This is urgent and fast-paced work with far-reaching implications for the company and society,” the preparatory post said.
Any tips? To contact this reporter via email, please specify the following address: scouncil@businessinsider.comor text, Signal, Telegram, or WhatsApp at 415-757-8198. Use a personal email address, non-work WiFi network, and non-work device. Here’s a guide to sharing your information securely.