This AI paper proposes COLT5: A New Model for High-Quality and Fast Long-Range Input Employing Conditional Computation

Machine learning models are needed to encode long-form text for various natural language processing tasks, such as summarizing long documents and answering questions. Using the Transformer model to process long texts is computationally expensive because attention cost increases quadratically with input length, and feedforward and projection layers must be applied to each input token. increase. Recently, several ‘efficient transformer’ strategies have been published to reduce the cost of attention mechanisms for long inputs. Nevertheless, feedforward and projection layers (especially for large models) carry most of the computational load and can make parsing long inputs impossible. In this work, we introduce a new model family, COLT5, which builds on LONGT5 and enables rapid processing of long inputs by integrating architectural enhancements of both attention and feedforward layers.

The foundation of COLT5 is the understanding that certain tokens are more important than others, and that by allocating more computing to important tokens, we can get higher quality while reducing costs. For example, COLT5 splits each feed-forward layer and each attention layer into a light branch that applies to all tokens and a heavy branch that is used to select important tokens specifically chosen for its inputs and components. Separate. Compared to regular LONGT5, the hidden dimension of the light feedforward branch is smaller than that of the heavy feedforward branch. Also, the proportion of significant tokens decreases with document length, making it easier to process long texts.

An overview of the COLT5 conditional mechanism is shown in Figure 1. Thanks to COLT5, two more changes have been made to the LONGT5 architecture. The heavy-attention branch exercises full attention across a different set of carefully-chosen significant tokens, while the mild-attention branch has fewer heads and applies local attention. Multi-query cross-attention, introduced in COLT5, dramatically speeds up inference. Additionally, COLT5 uses the UL2 pre-training target, which has been shown to enable in-context learning across long inputs.

Researchers at Google Research propose COLT5, a new model for remote input that uses conditional computing to improve performance and speed up processing. They demonstrated that COLT5 outperforms LONGT5 on the arXiv summary and TriviaQA question answering datasets, outperforms LONGT5, and reaches SOTA on the SCROLLS benchmark. COLT5 significantly improves the quality and performance of jobs with long inputs through non-linear scaling of the “focus” token. COLT5 also performs significantly faster fine-tuning and inference with equal or better model quality. COLT5’s light feedforward and attention layers are applied to all inputs, while heavy branching only affects the selection of significant tokens chosen by the learned router. They demonstrated that COLT5 outperforms LONGT5 on various long input datasets at all speeds and can successfully and efficiently use very long inputs up to 64,000 tokens.

Please check paper. All credit for this research goes to the researchers of this project.Also, don’t forget to participate 26,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more.