Visually-enabled artificial intelligence (AI) medical scribes have the potential to improve the accuracy of patient records and save clinicians valuable time. Getty Images
Incorporating vision-enabled artificial intelligence (AI) into medical scribes (recording devices that doctors use to document patient interviews in real time) has the potential to improve the accuracy of patient records and save clinicians valuable time.
A study from Flinders University has been published. npj digital medicine, It turns out that while AI medical scribes are already relieving some of the administrative tasks that take up patients’ time, they have the ability to do even more by attaching visual recorders to these devices.
Researchers at the Flinders School of Medicine and Public Health found that vision-enabled AI writing, employing a combination of Google’s Gemini model and Ray-Ban Meta smart glasses, significantly improved the accuracy of documenting pharmacist and patient encounters and reduced omissions and errors in clinical records.
“AI scribes are already listening to consultations and assisting clinicians, but there is more to medicine than just the spoken word,” said study author Bradley Mentz, an academic pharmacist at Flinders School of Medicine and Public Health.
“Much clinically important information is visual. Important visual cues during a consultation include a patient’s medication containers, prescriptions, equipment, and body language. When an AI system can use both what is heard and seen during the consultation, it can capture more details that are important to patient care.”
In the study, 10 clinical pharmacists recorded 110 “mock” medication history interviews, which included containers of more than 100 different medications, including tablets, capsules, injections, and creams.
The researchers recorded the interviews wearing Meta AI’s Ray-Ban glasses and passed the video footage to an AI scribe developed using Google’s Gemini AI model.
Analyzing both video and audio, the AI scribe achieved 98 percent accuracy, compared to 81 percent when the same system processed only audio information.
A key advantage was the ability to capture the strength and shape of the drug, details important for safe administration. AI scribes with video input captured this information 97% of the time, while audio-only recordings dropped to 28%.
“This is an enhanced tool and not a substitute for clinical judgment,” Mentz said. “Clinicians still have to review and approve the documentation.
“AI scribes can include verification steps, taking screenshots of drug packages and generating full audio transcripts, giving healthcare professionals a stronger foundation to review what the AI is producing.”
Senior author Associate Professor Ashley Hopkins said the study could mark the next stage in the use of AI scribes in the medical field.
“AI scribes are gaining traction because they reduce the burden of documentation and allow clinicians to spend more time with patients. These findings suggest that if scribes can see and hear, the next step will be to produce more accurate and complete drafts,” said Associate Professor Hopkins. “This means we spend less time editing AI documents and more time focusing on patient care.
“These findings suggest that the next step could be for all writing systems to be able to interpret visual as well as audio information. This could open the door to broader clinical applications.”
The authors say the study has some limitations, highlighting the need for human oversight and careful governance before these tools are more widely adopted. The paper also highlights privacy, consent, data security, and workflow integration as key issues that need to be addressed as vision-enabled AI scribes move closer to commercialization.
Paper – *Vision-enabled AI scribe reduces omissions in clinical conversations: Evidence from simulated drug histories, Bradley Mentz, Nicholas Scarfo, Natanshu Modi (University of South Australia), Eric Cornelis, Lee Li, Jin Kuan Eugene Tan, Jimit Gandhi (University of South Australia), Dosa Maher, Dev Kousa, Kezia Daniel, Vidya Menon, Stephen Bakki, Ross McKinnon, Michael Wiese (University of South Australia), Andrew Rowland, Michael Sorich, Ashley Hopkins -. npj Digital Medicine (2026). https://doi.org/10.1038/s41746-026-02494-9 *Please note that this is an unedited version of the manuscript to give you early access to the findings. The manuscript will be further edited before final publication. Please note that there may be errors that affect the content and all legal disclaimers apply..
Acknowledgment: The BDM PhD scholarship is supported by the Australian National Health and Medical Research Council (APP2030913). AMH is an Australian National Health and Medical Research Council Emerging Leaders Researcher Fellow (APP2008119). MJS is supported by a Beat Cancer Research Fellowship from the Cancer Council of South Australia. SB is supported by a Fulbright Scholarship.
