AI and algorithms are increasingly being used to make life-changing decisions in medicine and law, from assessing a patient’s cancer risk to determining criminal sentences. But how can we ensure that these algorithms are designed to remove, rather than reproduce, the biases inherent in human decision-making?
Emma Pearson, an assistant professor of computer science at the University of California, Berkeley, does this every day.
“Algorithms have many potential advantages over humans in being unbiased decision makers, but those advantages are not always realized, and it can be very difficult to achieve them,” Pearson said.
Mr. Pearson develops AI and machine learning methods for medicine and the social sciences with the goal of improving health care and reducing social inequalities. Recently, she developed a new test to detect racism in law enforcement. She also works to reduce racial and gender bias in algorithms, particularly those used to assess disease risk and make important decisions about patient care.
“I’m using AI to create a healthier and more just world,” Pearson said.
Pearson was recently named Zhangjia Endowed Professor and is affiliated with the Berkeley AI Institute, Computational Precision Health, and the Center for Human-Compatible AI.
more human than particles
Pearson was an undergraduate studying physics at Stanford University when she learned that her mother, who had been diagnosed with breast cancer several years earlier, was a carrier of the BRCA1 gene mutation. Ms. Pearson also has a 50% chance of carrying this mutation, which increases her risk of breast and ovarian cancer. Genetic testing several years later confirmed she was a carrier.
“I was very scared and upset, but I was very focused on finding what I wanted to do with my life,” Pearson said.
Pearson, who used statistical methods to study galaxies, already had the idea that he was more interested in humans than particles. She decided to turn previous research on its head and focus on how computer models can be applied to medical and social science fields.
Shortly after starting a graduate program in computer science, also at Stanford University, Pearson began studying police traffic stop data collected as part of the Stanford Open Policing Project. This data revealed undeniable racial disparities and inspired Pearson to focus not only on improving health care, but also on identifying and addressing racial inequities.
“When you look at the very basic statistics in the data, the racial disparities in the likelihood of being searched and arrested after being stopped, the disparities were huge,” Pearson said. “It was clear that something catastrophic was happening.”
But her research on police traffic stop data also highlighted the importance of designing accurate statistical tools to understand how and why inequalities arise. In a study published late last year, Pearson and her collaborator, Cornell University graduate student Nora Guerra, devised a new method to test whether inequality is caused by discrimination rather than other factors.
I’m using AI to create a healthier and more just world.
emma pearson
Traffic stop data from Texas, Colorado, and Arizona found that police found a higher number of people who reported being white in some encounters and Hispanic in others. The study compared how these people were treated if they were considered white versus Hispanic, and found that drivers were significantly more likely to be searched and arrested if police identified them as Hispanic.
The findings provide strong evidence that racial bias is likely the cause of the disparity.
“We focused on white and Hispanic drivers because they were the racial groups most likely to be confused in the specific data we looked at, but there’s no reason it couldn’t be applied to other categories,” Pearson said.
This same approach can be applied when a person’s racial or gender category is determined by how they are perceived rather than self-reported, or when multiple perceptions of the same person exist over time.
“For example, if race is not consistently recognized, we can look at the same person over time in different medical settings and see how they are treated differently,” Pearson said.
counter human prejudice
Using data to identify the root causes of racial and gender disparities is only the first step in addressing these disparities. Pearson is also working on designing AI algorithms that can help counteract the biases inherent in human thinking.
However, creating and implementing fair and unbiased algorithms in the real world presents many challenges. Just like humans, algorithms trained on biased data will make biased decisions. Additionally, many of the algorithms used in medicine and law are created by companies and are proprietary, making it difficult for the public to scrutinize or criticize how the algorithms were designed and trained.
“If algorithms are unfair, they have the potential to reproduce unfairness on a much larger scale than any single human decision maker,” Pearson said.
But Pearson points to a New York Times article by Massachusetts Institute of Technology professor Sendhil Mullainathan that argues that “algorithmic discrimination is easier to detect and correct than human discrimination.” Experiments can be designed to reveal biases even in proprietary algorithms, and researchers like Pearson are trying to find ways to address these biases once they become apparent.
Recently, Pearson has been researching how to design algorithms that perform well even in the presence of missing data. This problem is especially true in medicine. In medicine, algorithms that predict disease risk often omit many important factors (such as a patient’s genetic history or exposure to pollutants) because they are unknown. Instead, the patient’s race is often used as a variable.
“Race is a socially constructed variable, not a biological one,” Pearson says. “And historically, race has been factored into medical decision-making and algorithms in racist ways. By using race in these algorithms, we may worry that health disparities are further entrenched.”
If algorithms are unfair, they can also reproduce unfairness on a much larger scale than a single human decision maker.
emma pearson
However, simply removing race from medical algorithms will not necessarily reverse health disparities. In a 2024 paper, Pearson and her team found that removing race from cancer risk prediction algorithms could lead to underpredicted risks for black patients and reduced access to colorectal cancer screening. And in a research study published last month in JAMA Internal Medicine, Pearson and colleagues found that most patients are comfortable with medical algorithms including race as long as doctors are transparent about how the information is used.
“I think we could say that in a world where we had access to all the data on all the people and everything was completely fair, there would be no need to use race in algorithms,” Pearson said. “But it’s also important to consider the world we actually live in and use the data we actually have to design algorithms that help us make the best decisions for patients today.”
Pearson said doctors decided to test her mother for the BRCA1 gene because she is Ashkenazi Jewish, an ethnicity that is 10 times more likely than the general population to carry these cancer-causing mutations. Without testing, Pearson’s mother may not have lived to see her children grow up, and Pearson may not have had this important information about his medical risks.
“My experience [with the BRCA1 mutation] “It became really clear to me that these risk tools are not just abstract concepts or interesting research subjects, but have tangible impacts on people’s basic health and life decisions. It’s really important to get these tools right,” Pearson said.
