Researchers in the School of Engineering and Applied Science have launched Observer, the first multimodal healthcare dataset that captures anonymized, real-time interactions between patients and clinicians, giving outsiders a peek inside primary care clinics.
Demonstrating the recording process with Kevin Johnson (left) and Karen O’Connor (right).
(Image: Sylvia Chan)
Until now, the data available to medical researchers has been limited to the traces left after a visit, including qualitative information such as clinician notes and quantitative measurements such as patient vital signs. These sources do not capture subtleties such as body language and tone of voice, as well as environmental factors such as computer use that influence how providers and patients interact.
“Much of what shapes medical visits and their outcomes is invisible to researchers,” says Kevin B. Johnson, the David L. Cohen University Professor and lead author of a new paper describing observers. Journal of the American Medical Informatics Association. “Thanks to technology that enables HIPAA compliance and anonymizes recordings, Observer is able to observe care as it unfolds. This type of evidence is not only the foundation for improving clinical practice, but also essential for developing responsible AI tools to enhance care.”
Researchers have already begun using Observer by awarding pilot grants to other teams with the aim of expanding the dataset into a national resource for improving health care. “These early projects are the beginning of the flywheel,” Johnson says. “As researchers generate new insights and records, the dataset grows and allows us to ask even more ambitious questions.”
Observer links video, audio, and transcripts to clinical data and electronic health records, allowing researchers to ask new questions such as: How often clinicians see patients and computer screens. How room layouts and digital writing technology change communication. Patient response to explanation of diagnosis.
“This kind of multimodal evidence that combines video, audio, and medical records opens up opportunities in so many fields,” said Karen O’Connor, associate director of Johnson’s Artificial Intelligence Innovation Lab for Ambulatory Care. “By making this data available, we are opening new avenues to democratize medical research and improve care.”
For more information, please visit Penn Engineering.
