When it comes to adopting artificial intelligence in high-stakes environments such as hospitals and planes, short worker training on superior AI performance and technology is not sufficient to ensure that the system works smoothly and that patients and passengers are safe, new research suggests.
Instead, it is necessary to simultaneously evaluate the algorithms and those who use them in the most safety critical organizations.
The team also argues that these assessments need to assess how people respond to performance of good, mediocre, and poor technology, leading AI-human interactions to meaningful testing, and expose levels of risk linked to mistakes.
The participants in the study, led by engineering researchers at Ohio State University, were 450 nursing students from Ohio State, primarily undergraduates and 12 licensed nurses with varying amounts of clinical training. They used AI-assisted technology in remote patient monitoring scenarios to determine the possibility that emergency care would be required in a variety of patient cases.
Results showed that more accurate AI predictions about whether patients were trending towards medical emergencies resulted in 50% to 60% improvement in participants' performance. However, when the algorithm produced inaccurate predictions, human performance was disrupted, with explanatory data that did not support the results, with over 100% decomposition of the right decision when the algorithm was the most incorrect.
“AI algorithms are by no means perfect. So if you need an AI algorithm that is ready for a safety critic system, it means that teams, people, AI are coming together.
“The key is that this isn't really about creating a really good safety-critical system technology. What's important in a system where safety is important is the collaborative human machine functionality.”
Molly completed his research with Associate Professor Mike Raio and undergraduate honorary David Woods in both Ohio Integrated Systems Engineering. This study was recently published NPJ Digital Medicine.
The author, who is all members of the Cognitive Systems Engineering Lab, supervised by Rayo, developed a joint activity test research program in 2020 to address what is considered a gap in the deployment of responsible AI in high-risk environments, particularly in health and defense environments.
The team is also improving the set of evidence-based guided principles for mechanical design, with collaborative activities in mind that can smooth the AI and human performance assessment processes and then actually improve system outcomes.
According to their spare list, machines should first and foremost tell people how it is misaligned by the world, even when they don't know that they are inconsistent with the world.
“Even if the technology worked out with these heuristics, they probably aren't ready yet,” says Rayo. “These are risk mitigation procedures and industries that are critical of our safety need to do some form of empirical assessment, as they deserve at least these two steps to measure people and AI performance and examine a variety of challenging cases.”
Cognitive Systems Engineering Lab has been conducting five years of research into real-world technology to reach a practice assessment method, primarily a project with 20-30 participants. The 462 participants in the project, particularly the target population of AI injection techniques, where research enrollment is associated with course-based educational activities, gives researchers a high degree of confidence in their findings and recommendations, Rayo said.
Each participant analyzed a sequence of 10 patient cases under different experimental conditions. No AI help, AI percent prediction of the urgent need for emergency care, AI annotations of data related to patient status, and both AI prediction and annotation.
All examples included visualization of data showing demographics, vital signs, and lab results intended to help users predict patient status changes and stability.
Participants were instructed to report concerns for each patient on a scale of 0-10. Higher concerns about emergency patients and lowered concerns for non-urgent patients were indicators considered to indicate improved performance.
“We found that neither nurses nor AI algorithms are universally superior to other nurses in all cases,” the author writes. The analysis explained differences in participants' clinical experiences.
The overall results provided evidence that this type of assessment is necessary, but the researchers said they were surprised that the explanations included in some experimental conditions were rarely shaking with participants' concerns.
“No matter what effect these annotations had, I was roundly overwhelmed by the presence of indicators that have driven away everything else,” Rayo said.
The team considered research methods that include custom built technologies that represent the healthcare applications currently in use. This is a template for why recommendations are needed and how the industry can arrange proposed practices.
The coding data for the experimental technology has been published, and Morey, Rayo and Woods further explained their work in an article published on AI–frontiers.org.
“What we're arguing is how we help people better understand the various effects that can arise from technology,” Molly said. “Essentially, the goal is not the best AI performance. It's the best team performance.”
This study was funded by the American Nurse Foundation, which rethinks nursing initiatives.
