Mena Davidson • June 13, 2025
Professor Qu's research aims to reduce the accessibility and sustainability of high-performance generation AI, while reducing the cost and complexity of large-scale basic models of fine-tuning and training.
Qing Qu, an assistant professor in the Department of Electrical Engineering and Computer Science at the University of Michigan, has been named one of the recipients of the Google Research Scholar Award in Machine Learning. The award recognizes his pioneering project, “Learn Compressibility Dynamics for Adaptation & Training Basic Models.” This addresses the key efficiency challenges of training and fine-tuning basic models while providing strict mathematical assurance.
“The training and deployment of modern machine learning models requires important computational resources, raising concerns about GPU shortages and increased energy consumption, particularly with an empirical scaling method that suggests that exponential growth of model size and training data produces only minor performance improvements.” “The objective of this project is to address the challenges by leveraging the inherent low-dimensional structure of data and models to significantly improve the efficiency, adaptability, and scalability of large-scale underlying models of generated AI.”
Basic models such as Llama and Gemma, which form the backbone of modern natural language processing and visual applications, require large computational resources. QU's project aims to tackle these challenges head-on by leveraging mathematical insights into low-dimensional structures hidden in model weights and learning the dynamics to develop new low-rank fine-tuning and compression training methodologies.
QU will be built on his recent innovation “DeepLora” presented as an oral presentation in ICML'24 [1] Win and develop Best Student Poster Award at MMLS'24 It depends on the rankhow to fine-tune large-scale language models of low-rank adaptation (LORA) methods suitable for tasks. These techniques aim to simplify the hyperparameter tuning process and improve performance in low data scenarios. This study also explores training at the “Edge of Stability” (EOS). This is a regime that uses larger learning rates to promote better generalization and faster convergence. The second thrust uses a “blast matrix” (block level adaptive structure matrix) to develop a new training paradigm. [4]. These matrices go beyond traditional low-rank approximations and provide faster computation and better expressiveness.
“We are deeply grateful for Google's research for our generous support. Together, we will make high-performance generation AI more accessible and sustainable, reducing the cost and complexity of large-scale training models,” Qu said.
QU has published related research at several major machine learning conferences, including ICML [1],aistats [2]ICLR [3]New Rip [4]and Google Theory and Practice of Foundation Models Workshop. Together with his student and Professor Laura Barzano, he recently wrote a research paper [5] About related topics. His work has previously been recognized at the NSF Career Awards in 2022, the Amazon AWS AI Awards in 2023, and the UM Chinese Heritage & Scholarship Junior Fanculty Awards in 2025.
[1] Can Yaras, Peng Wang, Laura Balzano, Qing Qu. Compressible dynamics in deep overparameterized low rank learning and adaptation. 40th International Conference on Machine Learning, 2024 (Oral, Top 1.5%, MMLS Best Poster Award)
[2] Soo Min Kwon, Zekai Zhang, Dogyoon Song, Laura Balzano, Qing Qu. Efficient low-dimensional compression of overparameterized models. Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, pages 1009-1017, pp. 2024.
[3] Avrajit Ghosh, Soo Min Kwon, Rongrong Wang, Saiprasad Ravishankar, Qing Qu. Learn the dynamics of deep matrix factorization beyond the edge of stability. 13th International Conference on Learning Expression, 2025.
[4] Changwoo Lee, Soo Min Kwon, Qing QU, and Hun-Seok Kim. BLAST: Block-level adaptive structured matrix for efficient deep neural network inference. 2024 at the 38th Annual Meeting on Neural Information Processing Systems.
[5] Laura Balzano, Tianjiao Ding, Benjamin D. Haeffele, Soo Min Kwon, Qing QU, Peng Wang, Zhangyang Wang, Can Yaras. An overview of low-rank structures in training and adaptation of large-scale models. arxiv preprint arxiv: 2503.19859, 2025. (Authors are listed alphabetically)
