New AI tools address data accuracy and fairness to improve health algorithms

New York, New York [September 4, 2025]– A team of researchers from ICAHN School of Medicine in Mount Sinai has developed a new method of identifying and reducing bias in the dataset used to train machine learning algorithms to promote critical issues that may affect diagnostic accuracy and treatment decisions. The survey results were published in the online issue of September 4th Journal of Medical Internet Research [DOI: 10.2196/71757].

To tackle the problem, investigators have developed tools that help detect and correct bias in healthcare datasets before being used to train artificial intelligence (AI) and machine learning models. Investigators used various machine learning models to test aoquity of different types of health data, including medical images, patient records, major public health surveys, and national hygiene nutrition surveys. The tool was able to find both well-known and previously overlooked biases across these datasets.

AI tools are increasingly being used in healthcare to support decision-making, from diagnosis to cost forecasting. However, these tools are as accurate as the data used to train them. Some demographic groups may not be represented proportionally in the data set. Additionally, many conditions may be different or may be overdiagnosed between groups, investigators say. Machine learning systems trained with such data can perpetuate and amplify inaccuracies, miss diagnosis, and create suboptimal care feedback loops such as unintended outcomes.

“Our goal was to create actionable tools that help developers and healthcare systems identify whether bias exists in their data and take steps to mitigate it,” says first author Faris Gulamali. “We want to help these tools work well for everyone, not just the groups best represented in the data.”

The researchers reported that independence can adapt to a wide range of machine learning models, from simpler approaches to advanced systems that power large-scale language models. It can be applied to both small and complex datasets, and can evaluate input data such as lab results and medical images, as well as output such as predicted diagnosis and risk scores.

The findings of this study further suggest that independence is equally valuable for developers, researchers and regulators. It may be used during algorithm development, pre-deployment audits, or as part of a broader effort to improve healthcare AI equity.

“Tools like Epaiti are key steps to building a more equitable AI system, but they are just some of the solutions,” says senior author Girish N. Nadkarni, chairman of MD, MPH, Windreich, director of Artificial Intelligence and Human Health., Professors of Medicine Irene and Arthur M. Fishberg, the Icahn School of Medicine at Mount Sinai, and the top AI officers of the health system at Mount Sinai. “If we want these technologies to truly serve all patients, we need to pair technological advances with broader changes in the way data is collected, interpreted and applied. The foundation is important, and it starts with the data.”

“This research reflects an important evolution of thinking about AI in healthcare, not as a decision-making tool, but as an engine that improves the health of many communities we serve.” “By identifying and fixing inherent biases at the dataset level, we address the root of the problem before it affects patient care. This is a way to build wider community trust in AI and ensure that the resulting innovations improve not only what is best represented in the data, but also the outcomes of all patients.

The paper titled “Subgroup Learning: Detect, Characterize and Mitigate Implicit and Explicit Racial Bias in Healthcare Data Sets with Algorithm Development and Validation Research.”

The authors of the studies described in the journal include Faris Gramali, Ashwin Schliekant Sawant, Lora Liharska, Carol Horowitz, Lili Chan, Patricia Kobacci, Ira Hofer, Curundep Singh, Linne Richardson, Emmanuel Menser, Alexander Charney, David Reich, Gaianin Han,

This research was funded by the National Centre for Advancement in Translation Science and the National Institutes of Health.

For Mount Sinai Artificial Intelligence news, visit://icahn.mssm.edu/about/artificial-intelligence.

– ####-

About AI and Human Health Bureau at Mount Sinai

Leaded by Girish N. Nadkarni, MD, MPH – International authority on the safe, effective and ethical use of AI in healthcare – The Windreich division of AI and human health are pioneering the pioneering advancements at the US medical school, the intersection of artificial intelligence and human health.

The department is committed to leveraging AI in a responsible, effective, ethical and safe way to transform research, clinical care, education and operations. By bringing together world-class AI expertise, cutting-edge infrastructure and unparalleled computing power, the department streamlines the pathway for rapid testing and translation into practice while moving forward with breakthroughs in multi-scale, multimodal data integration.

The division benefits from dynamic collaborations across Mount Sinai, including Mount Sinai's Hasso Pratner Institute for Digital Health. This complements its mission by moving forward with a partnership between the Hasso Pratner Institute for Digital Engineering in Potsdam, Germany, and Mount Sinai Health Systems and a data-driven approach to improving patient care and health outcomes.

At the heart of this innovation is the famous ICAHN School of Medicine on Mount Sinai. It serves as a central hub for learning and collaboration. This unique integration enables dynamic partnerships between laboratories, faculties, hospitals and outpatient centers, driving progress in disease prevention, improving treatment of complex diseases, and improving quality of life on a global scale.

In 2024, the department's innovative Nootriscan AI application, developed in collaboration with faculty in the department developed by the Mount Sinai Health System's Clinical Data Science Team, has acquired Mount Sinai Health System. Nutriscan is designed to promote faster identification and treatment of malnutrition in hospitalized patients. This machine learning tool improves malnutrition diagnosis rates and resource utilization, demonstrating the impactful application of AI in healthcare.

For more information about the Windreich Department of AI and Human Health in Mount Sinai, visit ai.mssm.edu.

About Hasso Prattner Institute on Mount Sinai

At Hasso Pratner Digital Health Laboratory in Mount Sinai, tools of data science, biomedical and digital engineering, and medical expertise are used to improve and expand lives. The institute represents the collaboration between Hassoplatner Digital Engineering Institute in Potsdam, Germany and Mount Sinai Health System.

Under the leadership of Girish Nadkarni, MD and MPH, who leads the Institute, Professor Lothar Wieler, a globally recognized expert in public health and digital transformation, will co-oversee the partnership and change the way people think about personal health and health systems while driving positive innovation in patients' lives.

Mount Sinai's Hasso Prattner Institute for Digital Health is generously supported by the Hasso Prattner Foundation. Current research programs and machine learning efforts focus on improving our ability to diagnose and treat patients.

About Icahn School of Medicine in Mount Sinai

Mount Sinai's ICAHN School of Medicine is internationally renowned for its outstanding research, education and clinical care programs. It is the only academic partner of the Sinai Mountain Health System's seven member hospitals*, one of the largest academic healthcare systems in the United States, providing care to a large, diverse patient population in New York City.

Mount Sinai's ICAHN School of Medicine offers highly competitive MD, PhD, MD-PHD and Masters' Programs, with over 1,200 students enrolled. It is home to the country's largest graduate medical education program, with over 2,600 clinical residents and peers training throughout the healthcare system. The Graduate School of Biomedical Sciences holds 13 degrees, conducts innovative fundamental and translation studies, and trains over 560 postdoctoral research fellows.

Ranked 11th in the nation by National Institute of Health (NIH) funding, Mt. Sinai's ICAHN School of Medicine is one of the 99th percentiles of research funding per investigator, according to the American Association of Medical Colleges. Over 4,500 scientists, educators and clinicians work within and across dozens of faculties and interdisciplinary laboratories with an emphasis on translational research and treatments. Through Mount Sinai Innovation Partner (MSIP), Health Systems will promote the real-world application and commercialization of the medical breakthroughs that took place at Mount Sinai.

—————————————————————————————————————————————————————————————————————————————————————-

*Mount Sinai Health System Member Hospital:Mount Sinai Hospital. Mount Sinai Brooklyn; Mount Sinai Morning Side. Mount Sinai Queens; Mount Sinai South Nassau. Mount Sinai West; and New York Eye and Ear Treatment in Mount Sinai