ICML is world-renowned as one of the “big three” machine learning conferences, along with NeurIPS (Annual Conference on Neural Information Processing Systems) and ICLR (International Conference on Learning Representations). ICML brings together the world’s leading researchers to present and publish cutting-edge research on all aspects of machine learning, used in closely related fields such as artificial intelligence, statistics, and data science, as well as critical application areas such as machine vision, computational biology, speech recognition, robotics, and neuroscience. Professor Bapi Raju and PhD researcher Vijay Rowtula gave a poster presentation on their research titled “Linguistic properties and model scale in brain encoding: from small scale to compressed language models.”
what they did
“Since we can’t directly work on or examine the brain, we do the next best thing, which is to work on large-scale language models that exhibit the same capabilities as the brain,” explains Vijay. Their research began with assumptions first established by Richard Antonello’s scaling law, published in a 2023 NeurIPS paper. The assumption was that LLMs tend to represent language in a way that more accurately predicts human brain activity, and that going from small to large models would improve prediction accuracy by 15%. However, the IIIT-H researchers sought to disprove Antonello’s findings and instead worked on a smaller language model, demonstrating that it performs on par with LLM. In fact, they went a step further and tried to make the SLM smaller by introducing compression techniques such as quantization and pruning.
Quantization reduces the numerical precision used to represent information in the model. Pruning, on the other hand, removes parts of the model that are considered less important. Both approaches can reduce the memory and computational resources required to run an AI system. “It’s like decoupling neurons to make them lighter. What’s novel about our work is that we used multiple quantization techniques and used a framework to test the behavior of these models with respect to language,” elaborates Vijay.
what they found
Most of the compression techniques tested, such as quantization and moderate pruning, reduced the size of the model without significantly compromising the model’s ability to predict brain activity. The researchers found that models with about 3 billion parameters performed almost as well as larger models with up to 14 billion parameters. The study also found that while compression can weaken the model’s performance in language tasks including grammar, discourse, and morphology, its consistency with brain activity remains largely unchanged. “A new finding is that we observed a dissociation between brain tuning and language abilities for smaller and compressed versions of the language model. This indicates that the abilities needed to perform well on traditional language benchmarks may not be exactly the same representations that are important in modeling how the human brain processes language,” said Professor Raju.
A broader implication is that compact AI models may be sufficient to study language processing in the brain. Such models could make computational neuroscience research cheaper, faster, and easier to conduct without requiring researchers to work with the largest models available. “This could be a game changer for brain decoding workflows, which are important for designing brain-computer interfaces,” he said.
From computer vision to computational neuroscience
For Vijay, this work represents a new direction in an academic journey that has taken an unconventional path. He completed his master’s degree from IIIT-H in 2019 under Prof. CV Jawahar and is working on research in computer vision. He then spent several years in industry, including as a Principal Investigator, before returning to academia to complete his PhD. In computational neuroscience.
Moving from computer vision to studying language and the brain may seem like a big change. But Vijay sees broader relevance to the underlying computational model. “What I do is computational neuroscience,” he says. “We need to use computational models to mimic how the brain works.” His research also touches on a big question at the heart of the current AI race: How close can artificial systems get to human intelligence?
“Can computational models mimic the human brain?” he asks. “The whole idea of my PhD is to contribute to the current AI race to improve AI to human levels.” As AI systems increasingly move across domains, the distinction between verbal, visual, and other forms of intelligence is also becoming less clear. “There’s a fine line between the language model and the visual model. Under the hood, it’s always the same transformer,” says Vijay. His return to academia allows him to explore these questions from a different perspective, combining his experience working in industry with his background in computer vision.
