Recent developments in Large Language Models (LLMs) have demonstrated their superior problem-solving capabilities across several domains. LLMs can contain hundreds of billions of parameters and are trained on huge text corpuses.
Studies show that for LLM inference, memory bandwidth, not CPU, is the primary performance limitation for spawning tasks. This shows that in memory-constrained situations, the speed with which parameters can be loaded and saved rather than arithmetic operations becomes a significant delay barrier. However, advances in memory bandwidth technology have lagged far behind computational technology, resulting in a phenomenon known as the memory wall.
Quantization is a promising way to store model parameters with less precision than the usual 16-bit or 32-bit used during training. Despite recent advances such as LLaMA and its instruction-following variants, it is still difficult to achieve good quantization performance, especially for less bit-accurate and relatively modest models (e.g. 50B parameters). .
A new study from the University of California, Berkeley, delves into low-bit precision quantization and reveals the shortcomings of current methods. Based on these findings, the researchers introduced his SqueezeLLM, a post-training quantization framework that combines a dense-sparse decomposition technique with a unique sensitivity-based non-uniform quantization strategy. These methods enable quantization at ultra-low bit precision while maintaining competitive model performance, significantly reducing the cost of model size and inference time. Their method reduces the perplexity of the LLaMA-7B model at 3-bit precision from 28.26 with uniform quantization to 7.75 for the C4 dataset, which is a significant improvement.
Through comprehensive tests on the C4 and WikiText2 benchmarks, the researchers consistently demonstrated that SqueezeLLM outperforms existing quantization approaches across various bit precisions when applied to LLaMA-7B, 13B, and 30B for language modeling tasks. found to show significantly better performance.
According to the research team, low-bit-precision quantization of many LLMs is particularly difficult due to significant outliers in the weight matrix. These outliers affect the non-uniform quantization approach as well, as they bias the bit allocation towards extremely high or low values. A simple method is provided to split the model weights into dense and sparse components to remove outliers. By isolating the extrema, the central region shows up to 10 narrower ranges, improving quantization accuracy. Efficient sparse storage methods like Compressed Sparse Rows (CSR) allow you to keep sparse data with full precision. This method has low overhead by using efficient sparse kernels for the sparse part and parallelizing computations in parallel with the dense part.
The team demonstrates the framework’s potential quantized IF model by applying SqueezeLLM to the Vicuna-7B and 13B models. In their tests he compared the two systems. First, we assess the quality of the generated output using the MMLU dataset, a multitasking benchmark that measures model knowledge and problem-solving ability. We also use GPT-4 to rank the generative quality of quantized models compared to the FP16 baseline using the evaluation method presented in Vicuna. In both benchmarks, SqueezeLLM consistently outperforms his two current state-of-the-art approaches, his GPTQ and AWQ. It is worth noting that the 4-bit quantized model performs similarly to the baseline in both evaluations.
The study shows significantly reduced latency and improved quantization performance for models running on A6000 GPUs. The researchers demonstrated a speedup of up to 2.3 compared to baseline FP16 inference for LLaMA-7B and 13B. Moreover, the proposed method achieves up to four times faster latency than his GPTQ, demonstrating its effectiveness in quantization performance and inference efficiency.
please check out paper and github.don’t forget to join 24,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email me. Asif@marktechpost.com
featured tools From AI Tool Club
🚀 Check out 100’s of AI Tools at the AI Tools Club
Tanushree Shenwai is a consulting intern at MarktechPost. She is currently pursuing her bachelor’s degree at the Indian Institute of Technology (IIT), Bhubaneswar. She is a data her science enthusiast and has a keen interest in the range of applications of artificial intelligence in various fields. She is passionate about exploring new advances in technology and its practical applications.
