AMHERST — Since the advent of ChatGPT and similar AI models, university professors have been scrambling to stop students from coasting through assignments when chatbots seemingly do all the work for them. But how much does the use of AI actually help improve student performance?
Not much, according to a recent semester-long experiment at the University of Massachusetts Amherst aimed at examining the impact of AI on student learning. The experiment was led by Christian Rojas, professor and chair of the Department of Resource Economics.
“The inspiration had to do with the heated debates we see about the impact of AI on learning,” Rojas said in an interview with the Gazette. “I had also been investing in AI for a few years and wanted to get more involved in technology.”
Rojas, along with colleagues Ron Ron and Luke Bloomfield, conducted an experiment with students from two different time periods in the same class in an antitrust economics course taught by Rojas.
Students in the morning class were prohibited from using AI on assignments or in the classroom, while students in the afternoon received permission and structured guidance on using AI throughout the course.
Rojas said he chose to give afternoon classes access to the use of AI after observing that historically students in the afternoon section of the course performed slightly worse than students in the morning.
Students in both classes were told that they were participating in academic research, only that the research was about pedagogy and education, not about AI.
“The only difference between the two classes was the use of AI,” Rojas says. “We made sure they got the exact same lectures, slides, homework, exams, everything.”
However, when it came to the course’s two major exams and overall final grades, the professors found no significant differences between the two classes. The study found that traditional factors, such as a student’s cumulative grade point average, were far more effective in determining in-class academic performance than the use of AI.
Rojas said that while students may not have improved academically through the use of AI, they did see other signs of improvement when it came to studying. He noticed that students in his AI classes were more engaged with the subject matter, attended class more often, and were more active in class.
“I can’t say this here.” [study]“Because we don’t have a real variable to represent this, but as the semester progressed, just going to two classrooms was like two different experiences,” Rojas said.
In a survey distributed to students who took the course, students in the AI class also expressed a positive outlook toward technology and reported spending less weekly study time outside of class. I also noticed that students in my AI class were more likely to use AI outside of class to edit original work or check for grammatical issues, rather than using AI to get a complete answer.
“They didn’t do better on exams, but they spent less time studying,” Rojas says. “They get to the same goal line, but they can do it in a more efficient way and more aggressively.”
Of course, this study is only a small sample of the total number of students attending UMass, and an even smaller sample of college students nationwide, many of whom are likely using AI. But Rojas said the consistent patterns observed in how students interact in class depending on permission to use AI are certainly worth further discussion.
He said he would like to conduct an expanded version of such a study in the future, including tracking students during their first or second year to determine their subsequent performance in college.
“Seeing kids come up to me so enthusiastically and talk about AI and how to use it has given me more transparency,” Rojas said. “As an instructor, I told the class how we were using AI to help with specific tasks, so there was transparency on both sides, and I think that was really important.”
