Black box in healthcare artificial intelligence

introduction

From Face ID to Google Search, artificial intelligence is gaining popularity because it has the potential to greatly improve the efficiency and accuracy of systems. A subset of AI called “machine learning” is particularly used in the medical field. These machines, or deep learning models, are systems that are trained using relevant datasets to automate analytical processes. However, as adoption of this technology increases, concerns about understanding have arisen. This is called a “black box.” The black box metaphor refers to a box whose inputs and outputs are known without the context of the steps in between. Many doctors feel they don’t understand how machine learning systems work, which leads to the first problem: lack of transparency. Physicians also want to add biological logic to their algorithms instead of using a purely data-driven approach, but this leads to the second problem: program interpretability. As a result, calls for greater transparency in the application of such technologies are gaining momentum. This article reviews recommendations for physician transparency and interpretability in medical artificial intelligence applications.

method

While this technology has proven to be of great benefit to the industries that utilize it, the metaphorical black box in which it exists has raised questions, especially among physicians. This complaint suggests that healthcare providers do not have enough information or expertise to understand this technology effectively. This means that when working with AI models, clinicians are unable to answer a series of important questions, including:

• Can I confirm whether this model is suitable for the specific patients I am actually seeing?
• Can we trust that we will get reliable results to inform clinical decision-making?
• Does this model match what I already understand about human disease?
• What exactly did the model discover?

Although the list of approved algorithms continues to grow, healthcare providers are often in the dark when choosing an algorithm. Once doctors know what they want their AI systems to do, they access databases like the Data Science Institute AI Central and the FDA’s list of approved products to select programs that they think will work for them. If that doesn’t work, doctors refer back to the published list. All of this inefficiency stems from a lack of understanding among physicians, who are unable to choose a program that fits their needs.

To address this issue, the European Commission published the world’s first law aimed at regulating AI in April 2021. Under the Artificial Intelligence Act, an AI system was considered sufficiently transparent if the user could understand the AI ​​output and apply it accordingly. However, this outline is very vague and there are many different interpretations of this issue.

result

There are multiple perspectives on how to address this issue of lack of transparency. For the list of approved machine learning algorithms, Dr. Keith Dreyer, chief scientific officer of the American College of Radiology’s Data Science Institute, recommends that the FDA should standardize the information available in the catalog. He said metrics such as evaluation parameters, test demographics and findings can help physicians better understand a program before choosing one for their specific needs. This also helps eliminate inefficiencies and the risk of false negatives associated with choosing the wrong model.

Some experts suggest that in addition to changing the information available when choosing a system, the system itself needs to be changed to incorporate transparency measures. Poon and Sung elaborate on this idea using a model of black box interpretability. They argue that transparency can be increased in a step-by-step manner. In their model, a large amount of structural data is input into the algorithm. From there, the algorithm combines biological knowledge with its own computational knowledge in a learning process that can be monitored. The system accomplishes this through well-known tools such as Bayesian networks, fuzzy logic, and random forests. The system then enters the black-box stage, but Poon and Sung suggest the use of common interpretation techniques such as Garson’s algorithm, Lek’s profile, and the locally interpretable model-independent explanation (LIME) technique. This model aims to brighten the metaphorical black box that surrounds the steps in the system. As interpretability increases, doctors and patients can place greater trust in artificial intelligence.

Ordish et al. describe a system for detecting breast cancer risk as an example of how existing models can be modified with transparency in mind. For example, the new system will use heat map techniques to allow radiologists to focus on specific areas of the mammogram that the AI ​​will consider before making a decision. The system can also utilize standard text samples to justify its choices. This potential change could give the doctors performing the surgery greater insight into how the system itself is working. This modification idea aimed at transparency is being implemented by DeepMind at Moorfields Eye Hospital in London. Their machine learning model first prioritizes images of the patient’s eyes before analyzing the 3D scan of the eye. The system will be able to identify cases that require referral. To provide maximum transparency to the supervising physician, the program provides and evaluates multiple possible explanations for the decision and allows for visual identification of the part of the eye.

However, many experts dispute the idea of ​​pausing the use of machine learning, arguing that transparency is not essential. Eric Topol, director of the Scripps Research Institute’s Translational Institute, said, “If these models are validated in prospective clinical trials…there is every reason to bring them into the clinic with the hope of achieving a happy symbiosis between doctor and machine, even if doctors don’t know why the models work.” His opinion is echoed by many experts. They believe that while safety is paramount, physicians should not allow black boxes that prevent the use of trusted algorithms. As we highlighted earlier, there are many operational and potential machine learning programs that can, for example, capture subtle visual nuances that humans cannot discern. asks Regina Barzilai, a breast cancer deep learning modeler at the Massachusetts Institute of Technology. [artificial intelligence] Is it because our visual abilities are limited? ”

conclusion

We are in a reality where artificial intelligence systems are being widely implemented in medical settings. Although these systems have proven great potential to improve care, it is not always clear to healthcare providers and patients how such systems work. Some experts argue that the black boxing of medicine has necessitated modifications to increase interpretability. Some argue that blackboxing is not an inherent problem and can be ignored as long as the system is reliable and secure. This current debate is asking artificial intelligence developers to rethink how they create medical programs, while also encouraging healthcare providers and patients to reevaluate their priorities when considering treatment plans.

References

1. Bender, Eric. “Unraveling the Black Box of Medical Artificial Intelligence,” Undark Magazine, December 4, 2019. https://undark.org/2019/12/04/black-box-artificial-intelligence/.
2. Dryer, Keith. “Why Transparency in AI Medical Devices Matters,” www.acr.org, October 21, 2021. https://www.acr.org/Advocacy-and-Economics/Voice-of-Radiology-Blog/2021/10/21/Why-Does-AI-Medical-Device-Transparency-Matter.
3. Kiseleva, Anastasia, Dimitris Kotsinos, and Paul de Hart. “AI Transparency in Healthcare as a Multilayered Accountability System: Between Legal Requirements and Technical Limitations.” Frontiers of Artificial Intelligence 5 (May 30, 2022). https://doi.org/10.3389/frai.2022.879603.
4. Audish, Johan, Colin Mitchell, Hannah Murfett, Tanya Brigden, and Alison Hall. “Black Box Medicine and Transparency” PHG Foundation. University of Cambridge, https://www.phgfoundation.org/report/black-box-medicine-and-transparency.
5. Poon, Aaron IF, and Joseph J.Y. Song. Development of algorithms using artificial intelligence (AI) and machine learning. March 2021. Online images. Journal of Gastroenterology and Hepatology. https://onlinelibrary.wiley.com/doi/10.1111/jgh.15384#.
6. Poon, Aaron IF, and Joseph J.Y. Song. “Opening the black box of AI medicine” Journal of Gastroenterology and Hepatology 36, no. 3 (March 2021): 581–84. https://doi.org/10.1111/jgh.15384.
7. Zhang, Zhongheng, Marcus W. Beck, David A. Winkler, Bin Huang, Wilbert Sibanda, and Hemant Goyal. “Opening the Neural Network Black Box: How to Interpret Neural Network Models in Clinical Applications.” Annals of Translational Medicine 6, no. 11 (June 2018): 216–16. https://doi.org/10.21037/atm.2018.05.32.