Can artificial intelligence match a clinical interview assessment?

Clinical interviewing is one of the most important skills that physicians learn during their training. This forms the basis for accurate diagnosis and effective patient care. However, assessing these skills is often time-consuming and requires repeated observations and detailed feedback from experienced clinicians. As medical education continues to expand, this increased assessment burden has become a major challenge. The introduction of generative artificial intelligence (AI) could significantly improve the assessment of interviewing skills. However, its efficiency compared to standard rating systems is not well understood.

To fill this gap, Japanese researchers considered whether artificial intelligence could help solve this problem by evaluating medical interview records. Their findings were published in the journal Volume 12 on February 17, 2026. JMIR Medical Education. A research team led by co-author Dr. Hiromizu Takahashi and Professor Toshio Naito from Juntendo University School of Medicine’s Department of General Medicine investigated whether AI-based assessment (ABA) is comparable to traditional human-based assessment (HBA).

“Our core message is that AI can help make medical training fairer, faster, and more scalable.” Professor Naito explains:

To evaluate the ABA and HBA systems, researchers designed a cross-sectional validation study using a virtual patient system. Seven participants, including medical students, residents, and attending physicians, conducted clinical interviews with patients presenting with AI-simulated bilateral leg weakness. These conversations were automatically recorded and converted into transcripts. Records were then evaluated using the Master Interview Rating Scale, a standardized tool that assesses various aspects of clinical communication, including information gathering, organization, and empathy. For the ABA system, transcripts were evaluated using AI models, specifically GPT-o1 Pro and GPT-5 Pro. Meanwhile, five experienced clinical instructors independently evaluated the same transcripts comprising the HBA approach.

According to the researchers, ABA showed strong agreement with clinician ratings, with minimal differences in scores. At the same time, AI showed higher consistency across repeated assessments. Importantly, the use of AI also reduced the time required to evaluate each transcript by more than half, highlighting its potential to reduce educators’ workload. “Rather than replacing teachers, this study suggests a practical ‘AI-first, teacher-validated’ model where AI handles the first pass and educators focus their time on coaching, judgment, and high-stakes decisions.” says Dr. Takahashi.

These results have important implications for medical education. In many training programs, delayed feedback can limit students’ opportunities to improve their communication skills. By providing rapid and consistent assessments, AI can make iterative practice more accessible, especially in settings where faculty resources are limited. “Students can interview simulated patients with AI and receive near-instant feedback without having to wait days or weeks.” Professor Naito added, emphasizing the potential for more timely learning experiences.

At the same time, researchers stress that AI must be used with caution. Although the AI ​​performed well in this study, it was based on a small number of participants and a single clinical scenario. Additionally, transcript-based assessments fail to capture nonverbal cues, tone, and cultural nuances that are important in real patient interactions. Professor Naito and Dr. Takahashi caution as follows: “AI should be used under human supervision, as text-only scoring can miss nuances such as tone, nonverbal communication, and cultural background.”

Overall, this study highlights the growing role of AI in medical education. Combining the speed and consistency of AI with the expertise and judgment of clinicians has the potential to create more efficient and scalable training systems. As the demand for high-quality medical education continues to grow, such an approach could help ensure that future clinicians receive the best possible training while reducing the burden on educators.

***

reference
DOI: 10.2196/81673

Author: Hiromizu Takahashi¹Kiyoshi Shikino²Takeshi Kondo^3,4,Yuji Yamada⁵Yoshitaka Tomoda⁶Minoru Kishi⁷Yuki Aiyama⁸Sho Nagai⁹Akiko Enomoto⁹Yoshinori Tokushima¹⁰Takahiro Shinohara¹¹Fumiaki Sano¹Takeshi Matsuura¹²Rikiya Watanabe¹³Toshio Naito¹

Affiliation
¹Juntendo University School of Medicine, Department of General Medicine

²Department of Community Health Education, Chiba University Graduate School of Medicine

³Nagoya University Hospital Graduate School Clinical Training and Career Development Center (Nagoya City)

⁴Maastricht University, Faculty of Health Professions Education, Maastricht, The Netherlands.

⁵Brookdale Department of Geriatrics and Palliative Medicine, Icahn School of Medicine, Mount Sinai, New York, USA. ⁶Department of General Internal Medicine, Itabashi Central Medical Center

⁷Department of Internal Medicine, Nishiwaki City Hospital

⁸Tenri Hospital Anesthesiology/Emergency Department (Nara Prefecture)

⁹Department of Nursing, Faculty of Nursing, University of the Human Environment

¹⁰Department of General Medicine, Saga University Hospital

¹¹Department of General Medicine, Graduate School of Medical and Dental Sciences, Tokyo University of Science ¹²Hokkaido Bibai City Hospital General Medical Department

¹³Kitaharima Medical Center General Internal Medicine

About Professor Toshio Naito
Toshio Naito, MD, MBA, is a professor in the Department of General Medicine, Juntendo University School of Medicine, Tokyo. With over 30 years of clinical and academic experience, his research focuses on general medicine, infectious diseases, HIV, and medical education. He has authored 112 original articles and 4 review articles, achieving an h-index of 23 and 1,799 citations. His contributions led to significant advances in both clinical practice and medical training.