A team of researchers from the ICAHN School of Medicine in Mount Sinai has developed a new method of identifying and reducing bias in the dataset used to train machine learning algorithms to train critical problems that affect diagnostic accuracy and treatment decisions. The survey results were published in the online issue of September 4th Journal of Medical Internet Research [DOI: 10.2196/71757].
To tackle the problem, investigators have developed tools that help detect and correct bias in healthcare datasets before being used to train artificial intelligence (AI) and machine learning models. Investigators used various machine learning models to test aoquity of different types of health data, including medical images, patient records, major public health surveys, and national hygiene nutrition surveys. The tool was able to find both well-known and previously overlooked biases across these datasets.
AI tools are increasingly being used in healthcare to support decision-making, from diagnosis to cost forecasting. However, these tools are as accurate as the data used to train them. Some demographic groups may not be represented proportionally in the data set. Additionally, many conditions may be different or may be overdiagnosed between groups, investigators say. Machine learning systems trained with such data can perpetuate and amplify inaccuracies, miss diagnosis, and create suboptimal care feedback loops such as unintended outcomes.
Our goal was to create practical tools that would help developers and health systems identify if bias existed in their data. We want to help these tools work well for everyone, not just the groups best represented in our data. ”
Faris Gulamali, MD, First Author
The researchers reported that independence can adapt to a wide range of machine learning models, from simpler approaches to advanced systems that power large-scale language models. It can be applied to both small and complex datasets, and can evaluate input data such as lab results and medical images, as well as output such as predicted diagnosis and risk scores.
The findings of this study further suggest that independence is equally valuable for developers, researchers and regulators. It may be used during algorithm development, pre-deployment audits, or as part of a broader effort to improve healthcare AI equity.
“Tools like Aequity are key steps to building a more equitable AI system, but that's just part of the solution,” says senior author Girish N. Nadkarni, MD, MPH, chairman of Windreich of The Windreich of Human Health and director of Digital Health's Hasso Plattner Institute. of Mount Sinai Health System. “If we want these technologies to truly serve all patients, we need to pair technological advances with broader changes in the way data is collected, interpreted and applied. The foundation is important, and it starts with the data.”
“This research reflects the way we think about AI in healthcare, not as a decision-making tool, but as an engine that improves the health of many communities we serve,” says David L. Reich, MD, Chief Clinical Officer of Mount Sinai Health Systems and chairman of Mount Sinai Hospital. “By identifying and correcting inherent biases at the dataset level, we address the root of the problem before they affect patient care, a way to build wider community trust in AI and ensure that the resulting innovations are not only best expressed in the data, but also improve outcomes for all patients.
The paper is titled “Detect, characterize and mitigate implicit and explicit racial bias in healthcare datasets with subgroup learning potential: Algorithm development and validation studies.”
The authors of the studies described in the journal include Faris Gramali, Ashwin Schliekant Sawant, Lora Liharska, Carol Horowitz, Lili Chan, Patricia Kobacci, Ira Hofer, Curundep Singh, Linne Richardson, Emmanuel Menser, Alexander Charney, David Reich, Gaianin Han,
This research was funded by the National Centre for Advancement in Translation Science and the National Institutes of Health.
sauce:
Mount Sinai Health System
Journal Reference:
Grammar, F. , et al. (2025). Detect, characterize, and mitigate implicit and explicit racial biases in healthcare datasets with subgroup learning: algorithm development and validation studies. Journal of Medical Internet Research. doi.org/10.2196/71757
