California artificial intelligence (AI) heavyweights are slamming a state bill that would require tech companies to adhere to a strict safety framework, including creating a “kill switch” to halt powerful AI models, as a battle rages over regulating cutting-edge technology.
California lawmakers are considering proposals that would impose new regulations on tech companies operating in the state, including three of the largest AI startups, OpenAI, Anthropic, and Cohere, as well as large-scale language models operated by major tech companies such as Meta.
The bill, which passed the California Senate last month and is expected to be voted on in the state Assembly in August, would require a new state agency to provide assurances to California AI groups that they will not develop models with “dangerous capabilities,” such as creating biological or nuclear weapons or assisting in cybersecurity attacks.
The proposed Safe and Secure Innovation for Cutting-Edge Artificial Intelligence Systems Act would require developers to report the results of safety tests and implement so-called kill switches to shut down models.
But the law has drawn backlash from many in Silicon Valley, who argue it will drive AI startups out of the state and prevent platforms like Meta from operating under an open-source model.
“If you're going to create regulations that stifle innovation, you can't do much better,” said Andrew Ng, a prominent computer scientist who leads AI projects at Alphabet Inc.'s Google and China's Baidu Inc. and sits on the Amazon board of directors. “You're creating huge debt for science fiction risks and stoking fear in people who try to innovate.”
AI's rapid growth and huge potential have raised concerns about the safety of AI technology, with billionaire Elon Musk, an early investor in ChatGPT's developer OpenAI, calling AI an “existential threat” to humanity last year. This week, a group of current and former OpenAI employees published an open letter warning that “cutting-edge AI companies” are not receiving enough government oversight and pose “serious risks” to humanity.
The California bill was co-sponsored by the San Francisco-based nonprofit Center for AI Safety (CAIS), run by computer scientist Dan Hendricks, who serves as a safety adviser to Musk's AI startup xAI. CAIS has close ties to the Effective Altruism movement made famous by imprisoned crypto executive Sam Bankman Freed.
“Fundamentally, we want AI to succeed and continue to innovate, but let's work to avoid safety risks,” said state Sen. Scott Wiener, a Democrat who introduced the bill.
He added that the bill is “a lightweight bill that simply requires developers of large-scale models to conduct basic safety assessments to identify significant risks and take reasonable steps to mitigate those risks.”
But critics have called the Wiener bill overly restrictive, imposing costly compliance burdens on developers, especially smaller AI companies. Opponents also argue that the bill focuses on hypothetical risks, adding “extreme” liability risks to founders.
One of the fiercest criticisms is that the bill would negatively impact open source AI models, which are models that developers freely publish and build on top of their source code, such as Meta's flagship LLM, Llama. The bill would potentially make developers of open models liable for bad actors who manipulate their models to cause harm.
Arun Rao, lead product manager for generative AI at Meta, said in a post on X last week that the bill is “unworkable” and “will be the end of open source.” [California]. “
“The net tax impact of disrupting the AI industry and driving out companies could run into the billions of dollars as both companies and high-paid workers leave,” he added.
“This is the tech industry, and we don't like any regulation, so it's not surprising that there would be a backlash,” Weiner said of the criticism.
He said some of the responses were “not entirely accurate,” adding that he would be making amendments to clarify the scope of the bill.
The proposed amendments state that open source developers are not liable for models that have “gone through a lot of tweaking,” meaning that if an open source model becomes sufficiently customized by a third party, the group that created the original model is no longer liable. The amendments also state that “kill switch” requirements do not apply to open source models, he said.
Another amendment said the bill would only apply to larger models “that cost at least $100 million to train” and therefore would not affect most small startups.
“AI organizations are under competitive pressure that creates incentives to sacrifice safety,” CAIS's Hendrix said, adding that the bill is “realistic and reasonable” and that most people want “basic oversight.”
But senior venture capitalists in Silicon Valley said they're already fielding inquiries from founders asking whether the bill's passage will force them to leave the state.
“My advice to anyone asking is to stay and fight,” this person said, “but this will be a cold shower on the open source and startup ecosystem, and I think some founders will choose to leave.”
Over the past year, governments around the world have taken steps to regulate AI as the technology has become increasingly popular.
US President Joe Biden introduced an executive order in October aimed at setting new standards for AI safety and national security, protecting the public from AI privacy risks, and combating algorithmic discrimination, while the UK government outlined plans in April to introduce new laws to regulate AI.
Critics are perplexed by the pace at which California's AI bill has been passed by the Senate under CAIS's guidance.
The bulk of CAIS' funding comes from Open Philanthropy, a San Francisco-based charity with roots in the Effective Altruism movement. The organization has awarded CAIS roughly $9 million in grant funding in 2022-2023 aligned with CAIS' focus area of ”potential risks from advanced artificial intelligence.” The CAIS Action Fund, a division of the nonprofit founded last year, registered its first lobbyist in Washington, D.C., for 2023 and has spent about $30,000 on lobbying this year.
Weiner has received funding during several election cycles from Ron Conway, a wealthy venture capitalist who is a managing partner at SV Angel and an investor in AI startups.
Raed Ghani, a professor of AI at Carnegie Mellon University's Heinz College, said he saw an “overreaction” to the bill and that any legislation should focus specifically on use cases for the technology, rather than regulating the development of models.
