Even before testing of the model, GPT-4 Omni, began, OpenAI threw a party for employees at one of its San Francisco offices to celebrate the product that powers ChatGPT. “They planned a post-launch party before they knew it was safe to launch,” said one of the people, who asked not to be identified because the information is confidential. “We basically failed that process.”
The case, which has not been previously reported, highlights a shift in culture at OpenAI, where executives including CEO Sam Altman have been accused of prioritizing commercial interests over public safety, a stark departure from the company's roots as an altruistic nonprofit. It also calls into question the federal government's reliance on tech companies to self-regulate to protect the public from the abuse of generative AI, through White House pledges and an executive order on AI passed in October. Executives say generative AI has the potential to reshape nearly every aspect of human society, from work to warfare.
Andrew Strait, a former ethics and policy researcher at Google DeepMind and now deputy director of the Ada Lovelace Institute in London, said allowing companies to set their own safety standards is inherently dangerous.
Get caught up in
Stories to keep you up to date
“There is no assurance that internal policies are being adhered to or supported in a reliable manner,” Streit said.
Biden said Congress needs to enact new laws to protect Americans from the risks of AI.
“President Biden has been clear with tech companies about the importance of ensuring their products are safe, secure and trustworthy before they are released to the public,” White House press secretary Robin Paterson said. “Leading companies have made voluntary commitments around independent safety testing and public transparency, and the President expects the companies to meet them.”
OpenAI was one of more than a dozen companies that made a voluntary pledge to the White House last year as a precursor to an AI executive order. Others included Anthropic, maker of the Claude chatbot; $3 trillion semiconductor giant Nvidia; Palantir, a data analytics company that works with the military and government; Google DeepMind; and Meta. The pledge seeks to protect increasingly powerful AI models, and the White House has said it will remain in effect until similar regulations are enacted.
OpenAI's latest model, GPT-4o, was the company's first big chance to apply the framework, which calls for the use of human evaluators, such as post-doctoral experts trained in biology or third-party auditors, if the risk is deemed high enough. But despite complaints from employees, the testers cut the evaluation to one week.
While many employees had expected the technology to pass testing, they were dismayed to see OpenAI treating its vaunted new readiness protocols like an afterthought. In June, several current and former OpenAI employees signed a cryptic open letter calling on AI companies to exempt employees from non-disclosure agreements so they can warn regulators and the public about the safety risks of the technology.
Meanwhile, former OpenAI executive Jan Reicke, who resigned days after GPT-4o's launch, wrote to X that “safety culture and process have taken a back seat to flashy products,” and former OpenAI research engineer William Sanders, who left in February, said in a podcast interview that he had noticed a pattern of safety measures that were “rushed and not very robust” in order to “meet ship dates” for new products.
A representative from OpenAI's preparation team, speaking on condition of anonymity to discuss confidential company information, said the evaluation took place in a week, which would have been enough time to complete the tests, but acknowledged that the timing was “pushed in.”
“We are rethinking our entire approach,” the spokesperson said. “This is [was] It's just not the best way to do it.”
OpenAI spokesperson Lindsay Held said in a statement that the company “did not skimp on our safety process, but we recognize this launch has been stressful for the team.” To comply with the White House commitment, the company “conducted extensive internal and external testing” and initially withheld some multimedia features “as our safety efforts continue,” she added.
OpenAI announced the preparedness initiative as an attempt to bring scientific rigor to the study of catastrophic risks, which it defines as events “that could result in hundreds of billions of dollars of economic damage or cause serious harm or death to a large number of individuals.”
The term was popularized by a leading faction of the AI field who worry that attempts to build machines as smart as humans could weaken or destroy humanity. Many AI researchers argue that such existential risks are merely speculative and a distraction from more immediate harms.
“We aim to set a new high standard for quantitative, evidence-based work,” Altman wrote in an October post on X announcing the company's new team.
OpenAI launched two new safety teams last year, joining a long-standing division focused on specific harms like racial bias and misinformation.
The Super Alignment team, announced in July, was dedicated to preventing existential risks from highly advanced AI systems and has since been reassigned to other parts of the company.
The team was led by Reicke and OpenAI co-founder Ilya Sutskever, a former board member who voted to remove Altman as CEO in November but quickly retracted the vote. Both men resigned in May. Sutskever has been away from the company since Altman's return, but OpenAI didn't announce his departure until the day after the release of GPT-4o.
But an OpenAI representative said the team had the full support of senior executives.
Recognizing the tight timing for GPT-4o's testing, he met with company executives, including Chief Technology Officer Mira Murati, in April and agreed to a “backup plan”: If the evaluation uncovers any red flags, the company plans to release an earlier version of GPT-4o that the team has already tested.
A few weeks before the launch date, the team began “dry runs,” planning to have “all systems up and running as soon as the model was finalized,” the spokesperson said. In preparation for the test runs, the team arranged for human evaluators to work in various cities, a process that cost hundreds of thousands of dollars, the spokesperson said.
The preparatory work also included warning OpenAI's Safety Advisory Group — a newly created advisory board that will receive a scorecard of risks and advise leaders on if changes are needed — that it had a limited time to analyze the results.
OpenAI's Held said the company is committed to dedicating more time to the process in the future.
“We never [the tests]”It was a big deal,” the representative said, but he acknowledged the process was tough. “Afterwards, we said, 'We're never doing that again.'”
Razan Naqrawi contributed to this report.
