Mačina used MathTutorBench to test LLM for learning from OpenAI and Google, among others. This test revealed a significant difference. “We often see that there are trade-offs between different criteria. One model might be very good in terms of mathematical expertise, but not in terms of pedagogical ability; another model could be the other way around. Usually they can’t be balanced.” He says it’s also striking that many models lose their way at some point and veer off course when dealing with multi-level answers.
“It offers a better balance of specialized knowledge and teaching ability than a traditional LLM.”
In a second project by the same team, Mačina developed a unique LLM that aims for a better balance between pedagogy and didactics on the one hand and technical expertise on the other. He trained the model by having a virtual student interact with a virtual teacher in multiple steps, avoiding the use of expensive training data. This model learns from simulated interactions and feedback from a second model that monitors the teaching/learning process and evaluates the virtual teacher’s responses. Therefore, LLM continuously learns in a process known as reinforcement learning.
“The big advantage is that it doesn’t require huge amounts of data and can be done with a much smaller language model,” Machina explains. For comparison, OpenAI or Google’s modern LLMs have hundreds of billions to trillions of parameters. Simply put, the parameters are a measure of the LLM’s cognitive ability. Mačina’s model accommodates just 7 billion parameters.
“We find that our model has a better balance of technical expertise and teaching ability than a traditional LLM,” he added, adding that students are less likely to go off course, and even with a 20-step learning interaction, they don’t lose track of the course. During the learning process, you can also ask the model questions about the reasons for certain answers or decisions. “This allows teachers to track and monitor the learning process,” Machina said.
Will we soon see AI tutors for master’s students?
Mačina’s LLM is currently available for free under the name TutorRL and has already been downloaded more than 1,000 times. “To date, TutorRL is one of the few LLMs that is freely accessible and optimized for learning,” he says. However, he acknowledges that this model has not yet been tested and evaluated with learners in a classroom environment. To this end, he is currently looking for a partner at school. So far, this system only works for teaching mathematics at high school and early baccalaureate levels. However, in the long term, Mačina can certainly imagine this model being used in other STEM (science, technology, engineering and mathematics) subjects and becoming powerful enough for use at master’s level as well.
However, in his view, the results are not only relevant for education, but also have broader value for the further development of artificial intelligence. Collaborative problem solving like TutorRL will be essential in many fields of work in the future, as human judgment continues to be an important factor. “What we really want is a satisfying collaboration between a human and an LLM, not a model thinking for us,” says Mačina.
