When people seeking to become doctors submit applications to the Hofstra/Northwell Donald and Barbara Zucker School of Medicine, they hope that their academic achievements, volunteer work, and motivations will impress the first person to read about them. However, the first readers of these applications are not people, but artificial intelligence (AI) systems.
The AI system serves as an early screener for around 5,000 applications that come to schools in New York each year, recommending people who should be invited to interviews, those who do not meet baseline standards, and maves (large groups) to review for further consideration. Admissions Committee screeners review applications for those designated “yes” and “further reviews” and decide whether or not to receive an offer for an interview.
This process will help reviewers evaluate the application better, says Rona Woldenberg, the school's associate dean, for admission. “We took a pool of 5,000 applications and reduced it to 1,500-2,000 for committee review,” she says. “It made us more efficient and allowed us to focus our work where it really needed it” – consider carefully the most qualified applicants.
According to Woldenberg, using one AI tool means that all applications can remove many of the personal biases and variability that inevitably creep up between groups of human reviewers.
The Zucker School of Medicine is one of a few medical schools using or investigating the use of AI tools during enrollment. NYU Grossman School of Medicine in New York City also uses AI tools for its initial screening. The University of Cincinnati School of Medicine (UC School of Medicine) and George Washington University School of Medicine Health Sciences (GW SMHS) are developing AI platforms that they want to pilot within a year or two. Others, including the University of California, San Diego, and the School of Medicine, are debating how they do it.
Admission administrators at these schools believe that AI technology offers the potential to more efficiently and equitably handle a large number of applications exponentially beyond the slots available to students.
“Last year, we had 5,000 applicants and we have to get students who have accepted them to about 180. LaurahTurner, PhD, MS, Associate Dean of Artificial Intelligence and Educational Informatics at UC College of Medicine.
It requires a huge amount of staff resources. Kevin Nies of Med, admissions aide at GW SMHS, estimates that from July to February, each member of the review team spends 20-25% of the time assessing about 2,500 of the 13,000 applications. Research into the AI program at NYU Grossman School of Medicine estimates that the manual screening process for the application “includes more than 6,000 faculty hours per year.”
Admissions leaders discussed how AI improves some of the assessment processes, how leaders are developing AI programs to do so, and what they have done so far, and what they will do.
The advantages of AI
The School of Medicine provides enrollees with extensive training in how to assess applications, taking into account academic achievement, clinical experience, extracurricular activities, personal attributes, and school mission. Ideally, each reviewer evaluates all applications in the same way as other reviewers. In reality, that's not possible.
“With human reviewers, there will be a lot of variation in the perception of applicants,” Turner says.
One common variation is the value that reviewers on each committee potentially place even potentially in the student experience. Ioannis Koutroulis, MD, PhD, Associate Dean of MD Admissions at GW SMHS, points out that one reviewer may give more weight to applicants whose course and career ambitions are in the study, another reviewer may be dedicated to community service, and yet another may prefer those who agree with applicants from Harvard.
“It is inevitable that different reviewers will appreciate one experience more than another based on their background,” he says.
Even within an individual, when admissions judges remember through the application, shifts occur in perception and judgment. Turner asks rhetorically: “Is the processing of the first essay the same as the 500th essay?”
Then imagine a reviewer reading all the applications. Each applies the exact same criteria within seconds rather than 30 minutes. It could be AI.
“AI offers an approach that provides the same consistent review for all applications, potentially reducing the variation or subjectivity inherent in using a group of human screeners,” says Marc Triola, MD, dean of the Faculty of Education and Information Studies and professor of medicine at NYU Langone Health in New York City. Triola oversaw the AI pilot at the NYU Grossman School of Medicine.
Graham Keir is a neuroradiologist in Maryland, New York and AI consultant at Zucker School of Medicine. [information about the applicants]it always gives the same output. The clinical workload was not high that day. I'm not tired. ”
“What we're trying to do is be more efficient and fair.”
Its efficiency and fairness depends heavily on how AI systems are taught to evaluate applicants.
Teaching intelligence
How do you teach AI systems to learn what medical schools value most with future students? The admissions department partners with the Engineering and Technology department within the university system to build programs based on who the school has accepted in the past and what it wants in the application.
Baseline criteria are easy for AI to sort, such as minimum GPA and MCAT score. What's even more subtle is identifying experiences and statements that show which students best align with school priorities. Interest in primary care? Are you driving medical research?
“Download thousands of applications and train your system on what you're looking for,” says Koutroulis at GW SMHS. “After you tell the system, you want them to look for services, volunteers, jobs, public health” systems should learn how to identify them in the experience section of the application. [inputting] A long-standing application for a system to learn how to identify what you are looking for. ”
Identifying experiences and traits that match the school's mission is more than just searching for obvious keywords such as “service.” At UC College of Medicine, Turner is working on developing AI systems that can detect language patterns and potential properties. For example, an application with a reference to a public service may exhibit a quality of conscience, even though the applicant does not write “I am conscientious.”
The training process at the NYU Grossman School of Medicine illustrates a typical approach. Along with sampling of over 14,000 applications from 2013 to 2017 and the results of human screening for these applications, you can learn what the system is most likely to generate interview invitations. Researchers conducted trials of future students in subsequent admission cycles and saw how it was compared to the results of human reviews of those students. (AI ratings did not affect admission decisions during trial.)
The NYU Grossman School of Medicine and the Zucker School of Medicine report that pilot tests showed that AI recommendations repeatedly replicated recommendations from human reviewers. I am confident that the AI system correctly assesses applications according to school standards, so I have moved to using AI to perform initial screening.
These two schools use AI tools to evaluate the structured portion of the application. Applicants will provide facts about academic qualifications and extracurricular experiences. Cincinnati plans to begin evaluating essays only. Tools under development at GWU cover the entire application. These differences reflect what enrolled staff believe to provide the most useful initial outcomes for priorities as they take their first steps into AI assessment. They could evolve to use AI for other parts of your application.
The school follows guidelines developed by the committee convened by the association for the use of AI for protection against bias, alignment with medical school objectives, and data privacy.
future
The results of previous AI screenings identify cynical measures of success. It is to produce the same results that humans have produced. This saves a huge amount of staff time at the beginning of the review, but relying on data based on previous decisions makes previous biases permanent.
Admissions and technical engineers have not claimed that AI will remove bias from the selection process at this point. The results highlight the general challenges of employing AI to perform human-assigned tasks. If AI systems are built on past human outcomes, they reflect the human bias that produces those outcomes.
“When you train an AI model, there will be some bias,” Keir says. “How can I relieve that bias?”
For example, the system Keir tested with the Zucker School of Medicine aims to reduce bias by not providing information such as the applicant's name, birthplace, or photograph. Admissions and engineering staff hope to use AI to analyze micro-level decisions and see where long-standing biases have moved their ratings from or away from a particular applicant.
At GW SMHS, Nies hopes that AI will identify the characteristics of applicants who are most likely to end up enrolling in schools and become academically successful. “There are probably patterns and combinations we've never thought of before,” he says.
For all aspiring physicians, implementation of AI may raise concerns that their applications will be readily accepted or rejected by machines rather than by people. Over the years, many medical school computer programs have organized applications with basic standards, mostly academic scores, sending several applications to the effective “no” mountain, leaving them to be reviewed by the admissions committee.
Admissions instructors emphasize that AI tools will be reviewed by staff to make recommendations rather than decisions, ensure admission criteria are met, and who to invite to interviews.
“This is not a replacement for human reviews,” says Koutroulis.
