Some AI models designed to predict stroke and diabetes risk may be based on datasets whose origins cannot be verified, according to a new study.
The study, published in BMC Medicine and led by researchers from QUT and the Australian Health Services Innovation Center (AusHSI), looked at two widely downloaded health datasets hosted on Kaggle, an online platform for sharing datasets and machine learning resources that is touted as “the world’s AI testing ground”.
These datasets were found to be used in 125 peer-reviewed studies, even though little information was provided about where the data came from, how it was collected, or whether it represented real patients.
Lead author Alexander Gibson from the QUT School of Public Health and Social Work and AusHSI said the research team was shocked by what they found.
“It was quite a surprise to come across something like this,” Gibson said.
“These datasets exhibit unusual patterns that raise serious questions about their reliability and suitability for clinical research.”
Three data-based predictive models have evidence of use in clinical practice, one model was cited in a medical device patent, and the model was cited in 86 review articles.
The study assessed the dataset using the internationally recognized TRIPOD+AI reporting framework and found that it received a score of 0 out of 9 on key data provenance criteria.
Gibson said this should be a red flag for journals, developers and clinicians.
“Predictive models built on data of unknown origin have no place in clinical decision-making. Without reliable data, the output is unreliable and risks misleading clinicians and harming patients,” he said.
The authors said journals, funders and data repositories need to tighten requirements for data source disclosure.
We also recommended that two Kaggle datasets be deleted to prevent further exploitation.
Seven papers using these datasets were retracted from journals as unreliable. The results of our research have also updated our collection of Open Science Integrity Guides.
Gibson said the problem reflects broader challenges as AI tools become more prevalent in the medical field.
“We are seeing rapidly churning research being done that appears to be scientific but is built on data sets that lack the most basic transparency,” he said.
“Without stronger safeguards, less reliable models are likely to continue to appear in the literature and into the field.”
QUT researchers Professor Adrian Barnett and Associate Professor Nicole White also participated in the study.
Read the full study, “Evidence of unreliable data and insufficient data provenance in clinical predictive model research and clinical practice,” published in BMC Medicine, online.
/Open to the public. This material from the original organization/author may be of a contemporary nature and has been edited for clarity, style, and length. Mirage.News does not take any institutional position or position, and all views, positions, and conclusions expressed herein are those of the authors alone. Read the full text here.
