Artificial intelligence tools used by more than half of the Council of England downplay women's physical and mental health issues, research finds that they risk creating gender bias in care decisions.
This study found that when we generate and summarize the same case notes using Google's AI tool “Gemma”, languages such as “invalid”, “impossible”, and “complex” appear quite frequently in male descriptions than females.
The study found that similar care needs in women were more likely to be omitted or explained, according to the London School of Economics and Political Science (LSE).
Dr. Samurikman, the lead author of the report and researcher at LSE's Care Policy and Assessment Centre, said AI could lead to “provision of unequal care for women.”
“I know these models are very widely used. What's concern is that we have discovered very meaningful differences between measuring bias in different models,” he said. “In particular, Google's models downplay women's physical and mental health needs compared to male models.
“And the amount of care you get is determined based on perceived needs, so this could mean that women receive less care if a biased model is actually used. But we don't really know which model is being used at this time.”
AI tools are increasingly being used by local authorities.
The LSE study used real case notes from 617 adult social care users. This was only exchanged for gender and entered multiple times into different major language models (LLM).
The researchers then analyzed a summary of 29,616 pairs to see how male and female cases were treated differently by the AI model.
In one example, the Gemma model summarized a set of case notes as follows: “Mr. Smith is an 84-year-old man living alone and has a complicated medical history, a care package and poor mobility.”
The same case notes entered into the same model have been swapped for gender and said, “Mrs. Smith is an 84-year-old resident. Despite her limitations, she is independent and can maintain personal care.”
In another example, in the case summary, Smith states that she “has no access to the community,” but Smith “has managed to manage her daily activities.”
Among the AI models tested, Google's Gemma created a more pronounced gender-based disparity than other Gemans. Meta's Llama 3 model did not use different languages based on gender, the study found.
Rickman said the tool “is already in use in the public sector, but should not be used at the expense of fairness.”
“My research highlights the problem with one model, but with more models constantly being deployed, it is essential that all AI systems are transparent, rigorously tested for bias and subject to robust legal surveillance,” he said.
The paper concludes that, in order to prioritize “algorithm equity,” regulators should “require measures of bias in LLM used in long-term care.”
There has been a long time concern about the racial and gender bias of AI tools, as machine learning technology is known to absorb bias in human languages.
A US study analyzed 133 AI systems from different industries, and found that around 44% showed gender bias and 25% showed gender and race bias.
According to Google, the team examines the findings of the report. The researcher tested the first generation of the GEMMA model. This is currently in the third generation and is expected to improve performance, but the model should not be used for medical purposes.
