Guidance for safer AI-enabled medical devices: Dresden researchers emphasize the importance of the human factor

AI-enabled medical devices promise to improve medical care and support healthcare workers. However, the safety and performance of such systems does not solely depend on algorithms and technical specifications. How people use these devices and applications is equally important. In a paper recently published in a scientific journal NEJM AIa research team led by Professor Stephen Gilbert from the Else Kroner-Fresenius Center for Digital Health (EKFZ) at the TUD Dresden University of Technology is systematically analyzing the risks that may arise in human-AI interactions and making recommendations for manufacturers and regulatory assessors.

The authors show that existing regulatory approval requirements have so far only partially addressed many of these so-called “human factors-related risks.” This can create gaps that impact safety and quality of care. To address these, researchers identify seven key risks and develop practical recommendations for actions that can be integrated into existing regulatory and documentation processes.

Risks in using AI systems

AI-based medical devices can be used in a variety of areas in clinical settings. In radiology, for example, it helps detect cancer. Clinical decision support systems help choose the right treatment for the patient. In addition to real-time monitoring and early warning systems, AI can also support chatbots for patient communications and applications such as software that automatically generates medical reports or summarizes findings. This analysis focuses on the risks associated with the practical application of such AI systems. This includes, for example, that the opaque nature of AI systems increases the likelihood that their outputs will be misunderstood or misinterpreted. Problems can also occur if trust in an application is misaligned. As a result, users may over-rely on AI assistance or ignore relevant recommendations. The researchers also pointed to the risk of automation bias. This is the tendency to uncritically adopt recommendations from automated systems, potentially overlooking errors and relinquishing independent judgment. Additional risks include potential de-skilling, technostress among users, unchecked expansion of the display beyond its original intended scope (display creep), and errors related to system changes or different modes of operation. Such factors can cause additional burdens and unexpected failures in clinical practice, even if the technical performance of the system itself is good.

A practical guide for manufacturers and evaluators

For their analysis, the research team evaluated existing standards and regulatory guidelines for usability and safety, alongside scientific literature on AI in healthcare. Additionally, expert discussions from the fields of clinical applications, regulation, and human factors were incorporated. The result is a practical guide with seven recommendations that fill gaps in current standards. These are intended to support manufacturers and evaluators both before and after a product is brought to market. The aim is to early identify and systematically address AI-specific risks in interactions with human users.

The framework recommends developing and deploying AI-based medical devices in a way that clearly defines the users, the context in which the system will be applied, and which tasks are assigned to humans and which tasks are assigned to the system. Additionally, results should be presented in an easy-to-understand manner, integrated into existing clinical workflows, and supplemented, if necessary, with training and safe alternative options in the event of system failure. The authors emphasize the importance of continued monitoring after market entry. Usage patterns, potential misuse, or over-reliance on AI systems should be systematically observed and remediated as necessary. Changes to the system should also be communicated transparently so that work processes can be adjusted accordingly.

The recommendations are intentionally formulated in general and regulatory-aligned language to be applicable to a variety of AI-enabled medical devices and application scenarios. In the next step, the researchers aim to test and further develop their recommendations based on concrete pilot applications using AI-enabled medical devices. In the long term, human factors must be systematically considered in the regulation and evaluation of AI-based medical technologies to support safe innovation in healthcare while mitigating avoidable risks.

This article was written by researchers at the Dresden University of Technology (EKFZ for Digital Health, Head of Industrial Design Engineering, Faculty of Business and Economics) in collaboration with experts from the University of Oxford (UK) and the University Hospitals of Geneva (Switzerland).

publications

Rebecca Mathias, Anne Schmidt, Mateo Campos, Baptiste Vasey, Sebastian Lorenz, Peter McCulloch, Stephen Gilbert: Assessing human factors-related risks in AI-enabled medical devices: A practical guideNEJM AI, 2026. Link: https://ai.nejm.org/doi/full/10.1056/AIpc2501297

Digital Health at Else Kröner Fresenius Center (EKFZ)

The EKFZ for Digital Health at TUD Dresden University of Medicine and Carl Gustav Carruth Dresden University Hospital was established in September 2019. It is funded by the Else Kroner Fresenius Foundation for 10 years and approximately 40 million euros. The center focuses its research efforts on innovative medical and digital technologies that engage directly with patients. The aim here is to harness the full potential of digitalization in medicine to significantly and sustainably improve healthcare, medical research and clinical practice.