Visually-enabled artificial intelligence (AI) medical scribes have the potential to improve the accuracy of patient records and save clinicians valuable time. Getty Images
Introducing visual-enabled artificial intelligence (AI) to medical scribes, the recording devices doctors use to record real-time patient interviews, could improve the accuracy of patient notes and save clinicians valuable time.
A Flinders University study published in NPJ Digital Medicine found that while AI medical scribes already alleviate some of the administrative tasks that take up patients’ time, these devices have the ability to do even more by attaching visual recorders.
Researchers at the Flinders School of Medicine and Public Health found that vision-enabled AI writing, employing a combination of Google’s Gemini model and Ray-Ban Meta smart glasses, significantly improved the accuracy of documenting pharmacist and patient encounters and reduced omissions and errors in clinical records.
“AI scribes are already listening and assisting clinicians, but there is more to medicine than spoken words,” says study author Bradley Mentz, an academic pharmacist at Flinders School of Medicine and Public Health.
“Much clinically important information is visual. Important visual cues during a consultation include a patient’s medication containers, prescriptions, equipment, and body language. When an AI system can use both what is heard and seen during a consultation, it can capture more details that are important to patient care.”
In the study, 10 clinical pharmacists recorded 110 “mock” medication history interviews, which included containers of more than 100 different medications, including tablets, capsules, injections, and creams.
The researchers recorded the interviews wearing Meta AI’s Ray-Ban glasses and passed the video footage to an AI scribe developed using Google’s Gemini AI model.
Analyzing both video and audio, the AI scribe achieved 98 percent accuracy, compared to 81 percent when the same system processed only audio information.
A key advantage was the ability to capture the strength and shape of the drug, details important for safe administration. AI scribes with video input captured this information 97% of the time, while audio-only recordings dropped to 28%.
“This is an enhanced tool and not a substitute for clinical judgment,” Mentz said. “Clinicians still have to review and approve the documentation.
“AI scribes can include verification steps, taking screenshots of drug packages and generating full audio transcripts, giving healthcare professionals a stronger foundation to review what the AI is producing.”
Senior author Associate Professor Ashley Hopkins said the study could mark the next stage in the use of AI scribes in the medical field.
“AI scribes are gaining traction because they reduce the burden of document creation and allow clinicians to spend more time with patients. These findings suggest that the next step is for scribes to combine visual and auditory abilities to produce more accurate and complete drafts,” said Associate Professor Hopkins. “This means we spend less time editing AI documents and more time focusing on patient care.
“These findings suggest that the next step could be for all writing systems to be able to interpret visual as well as audio information, which could open the door to broader clinical applications.”
The authors say the study has some limitations, highlighting the need for human oversight and careful governance before these tools are more widely adopted. The paper also highlights privacy, consent, data security, and workflow integration as key issues that need to be addressed as vision-enabled AI scribes move closer to commercialization.
Paper – *Vision-enabled AI scribes reduce omissions in clinical conversations: Evidence from simulated drug histories, by: Bradley Menz, Nicholas Scarfo, Natansh Modi (University of South Australia), Erik Cornelisse, Lee Li, Jin Quan Eugene Tan, Jimit Gandhi (University of South Australia), Dorsa Maher, Dib Kousa, Kezia Daniel, Vidya Menon, Stephen Bacchi, Ross McKinnon, Michael Wiese (University of South Australia), Andrew Rowland, Michael Sorich, Ashley Hopkins published in npj Digital Medicine (2026). https://doi.org/10.1038/s41746-026-02494-9 *Please note that this manuscript is an unedited version to provide early access to the findings. The manuscript will be further edited before final publication. Please note that there may be errors that affect the content and all legal disclaimers apply.
Acknowledgments: The BDM PhD scholarship is supported by the Australian National Health and Medical Research Council (APP2030913). AMH is an Australian National Health and Medical Research Council Emerging Leaders Researcher Fellow (APP2008119). MJS is supported by a Beat Cancer Research Fellowship from the Cancer Council of South Australia. SB is supported by a Fulbright Scholarship.
/Open to the public. This material from the original organization/author may be of a contemporary nature and has been edited for clarity, style, and length. Mirage.News does not take any institutional position or position, and all views, positions, and conclusions expressed herein are those of the authors alone. Read the full text here.
