This article was originally published by CalMatters. Sign up for our newsletter.
A year ago, California officials were required to report how they used automated systems to make important decisions about people’s lives under a new state law.
They said that would never be the case, a surprising answer for a number of reasons, sources told CalMatters at the time, including some notable examples to the contrary.
Now, the state has released a broader response. The state currently uses six automated systems to make important decisions about Californians’ lives.
The system is used to do things like:
- Predicting whether incarcerated people will reoffend
- Assess whether unemployment claims are fraudulent
- Administer California State University student exams remotely
- Detect when college students create assignments using generative AI.
According to a report released on June 12th by the state’s technology bureau. The report is required under a 2023 law that requires state agencies to annually disclose their use of “high-risk automated decision-making systems,” which the law defines as systems “used to support or replace human discretionary decisions that have a legal or similarly significant impact, including decisions that have a significant impact on access to or approval of housing or accommodations, education, employment, credit, health care, or criminal justice.”
The law was pushed by civil rights, privacy, and civil rights groups concerned about harm from systems like AI. Many such systems have been shown to produce biased results against marginalized groups, such as those used in high-stakes tests, predicting recidivism, and detecting AI-generated text.
CalMatters found last year’s report alarming, noting that the state Department of Corrections used software to predict post-release behavior and that the Employment Department used a fraud detection system that suspended benefits for 600,000 Californians between Christmas and New Year’s 2020, according to a report by the Legislative Analyst’s Office.
The report lists six high-risk systems currently in use, some of which state agencies have been using for several years. These include COMPAS, which the California Department of Corrections and Rehabilitation has used for at least a decade to assign recidivism scores to inmates.
The technology division said in its report that more systems were included in the report this year because it more thoroughly evaluated state agency responses, including meeting with agencies and asking them questions about their systems.
In addition to the six high-risk systems, the department’s report also identified six additional systems that were initially reported as high-risk but were later determined not to be. One is the AI used by the California Department of Treasury to analyze bills.
The report also mentions two high-risk systems that are currently not in use. The Bureau of Cannabis Control is developing artificial intelligence to analyze whether marijuana packaging violates laws prohibiting appeals to children, and California State University has stopped using language models to review job applications.
The results of the second annual survey come after cities like San Jose and San Francisco released their first AI inventories in recent months. It also comes at a time when California-based AI companies such as Anthropic and OpenAI are going public and seeking government contracts. Americans are divided on whether they trust AI, and a survey last year by TechEquity and Carnegie California found that a majority of Californians want safety over innovation. A Gallup poll measuring Americans’ opinions found similar results.
Senate Bill 1248, a bill that would have prohibited state employees from using automated decision-making systems as the sole basis for making decisions, died in the state’s rapid-fire spending process last month.
what is missing
Although the newly released report shares more information than last year’s report, some questions remain about the state’s use of artificial intelligence and other automated systems.
According to the state’s website, the report does not include ongoing generative AI pilot projects supported by the governor’s office that help businesses pay their taxes or help state officials fight homelessness, or an AI assistant called Poppy that uses language models like Anthropic’s Claude to draft documents, conduct policy research, and build custom AI tools. Sixty-seven state agencies provided input during the pilot phase, and Poppy’s statewide rollout will begin next month, according to its website.
There is also no mention of California State University’s deal with OpenAI to provide a version of ChatGPT, although research into the use of AI in education has found that the technology is likely to do more harm than good.
A 2023 law requiring annual reporting of high-risk systems excludes reporting by many state agencies, including the judiciary and the University of California College System. A CalMatters report last month found that a majority of the roughly 60 courts operating across the state have adopted policies for the use of generative AI. Courts in Los Angeles and Riverside counties have begun testing an AI tool that acts as a clerk, writing orders and creating investigative notes.
CalMatters is compiling an inventory of automated decision-making systems used by state and local agencies across California to provide transparency into how government is using decision-making systems and AI. Do you know of an AI system being used by your state or local agency? Email khari@calmatters.org.
CalMatters is a nonpartisan, nonprofit news organization that brings Californians stories that explore, explain, and seek solutions to quality of life issues while holding leaders accountable.
