Study finds doctors often trust incorrect AI treatment recommendations

In an experiment in which doctors made decisions about the treatment of a hypothetical patient, they were more likely to trust inaccurate advice presented as being generated by artificial intelligence (AI), even after they had a chance to notice that the patient’s recovery data contradicted the recommendations. Aranzazu Vinas and colleagues at the University of the Basque Country in Spain are publishing these findings in an open access journal. PLOS Digital Health.

AI systems can help doctors categorize patients according to various care needs, such as whether a patient is likely or unlikely to benefit from a particular treatment. These systems are not perfect and are intended to be used as suggestions to help physicians spot and correct potential errors.

Previous research has shown that people generally have difficulty recognizing and correcting mistakes made by AI. To explore how this challenge extends to physicians, Vinas et al. analyzed data from 223 physicians who anonymously participated in an online experiment.

Physicians were asked to imagine that they had the option to treat patients with rare diseases using unproven treatments in development. They were told that the AI ​​system had identified which patients were more or less likely to benefit from the treatment. The doctors then selected which patients to treat and, after being presented with data about the patient’s recovery, rated their perceptions of how reliable the AI ​​was.

Importantly, the actual effectiveness of the hypothetical treatment did not match the AI ​​recommendations. In one experiment, the treatment had an equally moderate effect on all patients, whereas in a second experiment it had an equally ineffective effect on all patients.

However, in both experiments, physicians tended to rate the AI ​​systems as trustworthy and did not appear to use patient recovery data to conclude that the AI ​​recommendations were incorrect. In the second experiment, doctors were unaware that the treatment had no effect at all.

These findings highlight potential challenges in incorporating AI-based classification into healthcare. Future research could build on this work, including developing and testing strategies and protocols that can enhance human critical thinking and AI error detection to maximize the benefits of human-AI collaboration while minimizing potential errors. ”

“In both experiments, doctors primarily trusted the AI’s classification and struggled to learn from the feedback. Moreover, in the second experiment, the experts did not notice that the treatment was completely ineffective,” said lead author Aranzaz Vinas.

Co-author Helena Matute added: “Although it is often said that humans are always in control of algorithms, our experiments show that doctors (as well as everyone else) have trouble learning from the available evidence when it contradicts the algorithm’s recommendations.”

Co-author Fernando Blanco summarizes: “It is important to study the mistakes that humans (including doctors) make when working with algorithms and learn how to minimize the problems that arise from them.”