Opinion: AI risks could be devastating. Empower companies' workers to warn them

Editor's note: Lawrence Lessig is the Roy L. Furman Professor of Law and Leadership at Harvard Law School and author ofThey Don't Represent Us: Reclaiming Our Democracy“The views expressed in this commentary are his own. Read more opinion On CNN.

CNN
—

Daniel Kokotajiro resigned as a researcher at OpenAI, the developer of Chat GPT, in April, saying in a statement that he disagreed with the way the company was addressing security-related issues as it continues to develop revolutionary but poorly understood artificial intelligence technology.

Kokotajiro, who worked on policy and governance at OpenAI, expanded on his thoughts in a post on his profile page on the online forum LessWrong, writing that he left his job because he had “lost confidence in taking responsible action” to protect against potentially serious risks associated with AI.

And in a statement around the time of his resignation, he blamed the company's culture for forging ahead without heeding warnings about the dangers it might pose.

“They and others have embraced a 'move fast and break things' approach, which is the opposite of what's needed for such a powerful yet poorly understood technology,” Kokotajlo wrote.

OpenAI pressured him to sign a contract promising not to disparage the company, and told him that if he refused, he would lose his existing stake in the company, which The New York Times reported was worth $1.7 million, but he refused, apparently choosing to reserve the right to publicly voice his concerns about AI.

After Kokotajiro left OpenAI and reports emerged that the company had pressured him to sign a non-disclosure agreement, CEO Sam Altman quickly apologized.

“This is my responsibility,” Altman wrote on X (formerly Twitter). “This is one of the few times I've been truly embarrassed while running OpenAI. I had no idea this was happening, and I should have.” What Altman didn't disclose was how many of the company's employees and executives had been forced to sign similar contracts in the past. In fact, for years, former employees say, the company has threatened to revoke vested interests if employees don't promise to behave respectfully.

But Altman's apology was effective in reducing attention to OpenAI's legal blunders in mandating such agreements. The company was eager to move forward, and most media outlets were happy to oblige. Few reported the obvious legal truth: such agreements were patently illegal under California law. For years, employees had felt silenced by promises they felt forced to sign, but a humble apology from the CEO was enough for the media and the public to move forward.

We should pause to consider what it means that someone is willing to give up potentially millions of dollars to defend free speech. What exactly is he trying to say? And Kokotajiro, as well as many other recently resigned OpenAI employees, have expressed serious concerns about the dangers inherent in the company's technology.

I know Kokotajiro and was in contact with him after he left, and I currently represent him and 10 other current and former OpenAI employees pro bono, but the facts I present here are taken only from publicly available sources.

Many have cited concerns about this technology as a matter of “AI safety,” but that's a terrible term to describe the risks that many in the field are deeply concerned about. Some leading AI researchers, including Turing Award winner Yoshua Bengio and computer expert and neuroscientist Sir Geoffrey Hinton, sometimes called the “Godfather of AI,” worry that runaway systems are not just a “safety risk,” but could cause catastrophic damage.

While the average person cannot imagine how it would be possible to lose control of a computer (“just pull the plug!”), we should also recognize that we do not actually understand the systems that these experts fear.

Companies operating in the field of AGI (artificial general intelligence, broadly defined as the theoretical AI research that attempts to create software with human-like intelligence, including the ability to perform tasks that are not the subject of training or development) are some of the most lightly regulated and inherently dangerous businesses in America today. No agency has the legal authority to oversee how companies develop their technology or what precautions they are taking.

Instead, we rely on the good judgment of these companies to ensure that risks are appropriately managed. So as a handful of companies race to realize AGI, the most important technology of the century, we trust them and their boards to put the public interest first. What could go wrong?

This gap in oversight has led several current and former OpenAI employees to formally call on the companies to pledge to encourage an environment where employees are free to criticize the company's security practices.

Their Right to Warn pledge calls on companies to:

First, commit to removing any “non-disparagement” agreements. (OpenAI has already made a similar commitment, but there are reports that other companies may have similar language in their contracts that they have not yet acknowledged.)

Second, it asks companies to commit to creating anonymous mechanisms for employees and former employees to raise safety concerns to boards of directors, regulators, and independent AI safety organizations.

Third, it calls on companies to support a “culture of open criticism” and encourage employees and former employees to speak out about safety concerns, as long as the companies' intellectual property is protected.

Finally, and perhaps most interesting, while asking companies to promise not to retaliate against employees who share confidential information when raising risk-related concerns, if a company institutes a confidential and anonymous process, employees are committed to first airing their concerns through that process, intended to create incentives to build mechanisms that enable warnings while still protecting confidential information.

This “right to warn” is unique in U.S. corporate regulation. It is justified by the absence of effective regulation, but that could change dramatically if Congress were to address the risks many have identified. And the right to warn is necessary because regular whistleblower protections do not protect against unregulated conduct.

Laws, especially in California, give employees broad rights to report illegal activity, but with fewer regulations there is less illegal activity. Therefore, without effective regulation of these companies, only employees can identify the risks that companies are ignoring.

Even if an AI company supports the right to warn, current or former employees should expect it will not be easy to sue the AI ​​company. Whistleblowers are not favorite coworkers, even if they are respected by some. And even with formal protections, the choice to sue will inevitably affect future employment opportunities and friendships.

Clearly, it's unfair to resort to self-sacrifice to ensure that private companies don't prioritize profits over catastrophic risks — that's the job of regulation. But if former employees are willing to lose millions of dollars to be free to say what they know, maybe it's time for our representatives to build an oversight regime that makes such sacrifices unnecessary.