At a gathering One of the most talked-about topics at the Health Equity Leadership Conference this month was the benefits and risks of incorporating artificial intelligence in the health sector, which experts say may hold great promise for diagnostics and patient experience, but also seems like a lawless zone for the technology.
“As we know, there is racial bias built into the AI currently being used in patient care, and there is no clear responsibility or accountability to ensure that AI does not harm diverse populations,” Sheila Ock, chief engagement and equity officer at Lowell Community Health Center, said in introducing the panel at the Healthy Equity Trends Summit. “To get us on track, we need to focus on equity in every part of AI development and implementation.”
The panel, moderated by Rahsaan Hall, president and CEO of the Urban League of Eastern Massachusetts, explored practical applications and equity red flags of incorporating artificial intelligence into everyday work.
“As a primary care physician, it would be a dream to be able to sit down and talk to a machine and have it suggest diagnoses that might speed up treatment for the patient,” said Renee Crichlow, chief medical officer at Codman Square Medical Center. “It's a dream, but I don't think that's where it's going to have the biggest impact on patients.”
Instead, Crichlow said the benefit will be that providers will be able to use AI for operations rather than for individual patient diagnoses, allowing them to shift resources within their clinics.
“During the patient intake process, just by the patient's AI talking to our AI, it reduces the clinic's overhead by 30 percent. And you know what we're going to spend that money on? We're going to spend it directly on patient care,” Crichlow said.
Marzieh Ghassemi, an associate professor of electrical engineering and computer science at MIT, explained one possible use for the AI tool: A patient who is brought to a hospital with breathing difficulties to undergo a chest X-ray, but a doctor is not available for two hours, the idea is that the patient could be sent home if they are healthy.
Ghassemi said the public chest x-ray data set already contains more than 700,000 images, so his lab trained a neural network to predict “no abnormalities” for a given chest x-ray. In simple terms, they taught the machine to recognize potentially problematic findings in the scan, identify what the scan finds in healthy lungs, and recommend that the patient is healthy.
“There are tons of papers out there that claim that state-of-the-art clinical AI can perform as well as or better than humans,” Ghassemi says. The question is, what's going to happen, he says. Many papers, including his, report high accuracy and predictability of AI tools. So, “Should we adopt AI tools? To answer that question, I would say people are adopting AI tools. So it's not really a question. It's actually happening.”
These studies and technologies could be introduced into hospitals through a variety of routes: perhaps the Food and Drug Administration would review it and approve it for broad distribution, perhaps an Institutional Review Board would approve it, or perhaps hospitals could use it as an administrative aid without FDA or Institutional Review Board approval.
Even the most cutting-edge models are biased, Ghassemi said: An audit of a lung X-ray model found that the program was more likely to mistakenly exclude female, younger, black and Medicaid patients.
“If you're in an intersectional identity — if you're a black female patient or a Hispanic female patient — you're significantly worse off than if you're part of a larger collective group,” she said. “And you might say, that's a very specific example, surely this isn't that big of a problem. And it is.”
Microsoft and the health system Epic have partnered with Open AI, which runs the popular Chat GPT program, to create more than 150,000 medical records without FDA approval or institutional review board oversight. Demonstrating potential bias within the system, when Ghassemi changed the race of a “combative, violent” patient from white to black, the Open AI model changed from completing the record as “send to hospital” to “send to jail.”
Developing new models and algorithms should focus on making them less biased than older models, she said. Current risk scores, which calculate the expected cost of treating a patient, are based on biased data and are therefore “highly biased,” she said. For example, users should be aware that using a large group of data may produce stronger predictions overall, but may misdiagnose certain populations more often.
“In any field, but especially in healthcare, it's really important that organizational leaders make an effort to educate their staff, whether it's a receptionist using a system like Chat GPT to analyze notes, or a doctor using clinical algorithms to diagnose and treat, so that people understand what AI is and what it is not,” said Cade Crockford, director of the Technology for Liberty program at the ACLU of Massachusetts.
The governor's sweeping economic development bill highlights the potential of artificial intelligence, with a highlight being a $100 million investment in an Applied AI Hub.
When it comes to implementing artificial intelligence in healthcare, Ghassemi recommended that health systems take inspiration from areas where they are implementing safer technology integration, such as aviation, where pilots are required to undergo rigorous training to use automated tools.
“If we want to advance ethical AI in healthcare, it's important to consider sources of bias in the data and do more comprehensive assessments. We also need to recognize that we can't fix all the gaps — nothing is perfect, no model is perfect, no person is perfect. If we are careful about how we develop new tools, we can improve healthcare for all and move forward more equitably,” Ghassemi said.
