Introducing vLLM: An open-source LLM inference and serving library that speeds up HuggingFace Transformers by 24x

Large scale language models (LLM for short) have emerged as a breakthrough in the field of artificial intelligence (AI). These models, such as GPT-3, have completely revolutionized natural language understanding. With the ability of such models to interpret vast amounts of existing data and generate human-like text, these models will shape the future of AI and unlock new possibilities for human-machine interaction and communication. It has endless possibilities to open up. However, despite the great success achieved by LLM, one significant challenge often associated with such models is their low computational efficiency, leading to poor performance even on the most powerful hardware. As these models consist of millions and billions of parameters, training such models requires large amounts of computational resources, memory, and processing power that are not always accessible. Furthermore, these complex architectures with slow response times can make LLM impractical for real-time and interactive applications. As a result, addressing these challenges is essential if LLM’s full potential is to be realized and its benefits made more widely available.

Addressing this issue, researchers at the University of California, Berkeley, developed vLLM, an open-source library that is a simpler, faster, and cheaper alternative to inferring and serving LLM. The Large Model Systems Organization (LMSYS) is currently using this library to power Vicuna and Chatbot Arena. By switching the backend to vLLM, as opposed to the original HuggingFace Transformers-based backend, research organizations can efficiently handle peak traffic while using limited compute resources and reducing high operating costs. (5 times more than before) successfully processed. vLLM currently supports several HuggingFace models including GPT-2, GPT BigCode and LLaMA. It maintains the same model architecture and achieves 24x higher throughput levels than HuggingFace Transformers without any modifications.

As part of a preliminary investigation, researchers at Berkeley determined that memory-related issues pose a major constraint on LLM’s performance. LLM uses input tokens to generate attention key and value tensors, which are cached in GPU memory for generating subsequent tokens. These dynamic key-value tensors, known as KV caches, occupy a significant portion of memory, making their management a tedious task. To meet this challenge, the researchers introduced the innovative concept of PagedAttendance, a new attention algorithm that extends the traditional idea of ​​paging in operating systems to his LLM service. By storing key and value tensors in a non-contiguous memory space, PagedAttendance provides a more flexible approach to managing key and value tensors, eliminating the requirement for long contiguous blocks of memory. These blocks can be retrieved individually using the block table during attention computation, resulting in more efficient memory usage. Adopting this clever technique reduces memory waste to less than 4%, resulting in near-optimal memory usage. Additionally, PagedAttend can batch together 5x more sequences than his, resulting in better GPU utilization and throughput.

PagedAttendance offers the additional benefit of efficient memory sharing. During parallel sampling, that is, when multiple output sequences are created simultaneously from a single prompt, PagesAttention enables sharing of computational resources and memory associated with that prompt. This is achieved through the use of block tables. The block table allows different sequences within a PagedAttendant to share blocks by mapping logical blocks to the same physical block. By employing this memory sharing mechanism, PagesAttention not only minimizes memory usage, but also ensures safe sharing. Experimental evaluations conducted by the researchers revealed that parallel sampling reduced memory usage by as much as 55% and improved throughput by 2.2x.

In summary, vLLM effectively handles attention key and value memory management through the implementation of the PagedAttendance mechanism. This gives good throughput performance. Additionally, vLLM seamlessly integrates with the well-known HuggingFace model and can be used in parallel with various decoding algorithms such as parallel sampling. This library can be installed using a simple pip command and is currently available for both offline inference and online services.

please check out blog post and github.don’t forget to join 25,000+ ML SubReddits, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email me. Asif@marktechpost.com

🚀 Check out 100’s of AI Tools at the AI ​​Tools Club