Google DeepMind Introduces NaViT: A New ViT Model That Uses Sequence Packing During Training to Handle Inputs of Arbitrary Resolutions and Aspect Ratios

Vision Transformer (ViT) is rapidly replacing convolution-based neural networks due to its simplicity, flexibility, and scalability. The image is split into patches and each patch is linearly projected onto tokens, forming the basis of this model. Input photos are typically squared and split into a set number of patches before use.

A recent publication explores potential deviations from this model. Since FlexiViT allows a continuous range of sequence lengths, the costs are calculated by accommodating different patch sizes within a single design. This is achieved by randomly choosing patch sizes during each training iteration and using a scaling technique that accommodates a large number of patch sizes in the initial convolutional embedding. Pix2Struct’s alternate patching method that preserves aspect ratio is very useful for tasks such as graph and document comprehension.

NaViT is an alternative developed by researchers at Google. Patch n’ Pack is a technique that allows you to change the resolution while preserving the aspect ratio by packing many patches from individual images into one sequence. The idea is based on “sample packing”. This is a technique used in natural language processing to efficiently train models with inputs of varying lengths by combining multiple instances into one sequence. Scientists have found the following evidence:

Randomly sampling the resolution can significantly reduce the training time. NaViT delivers excellent performance across a wide range of solutions, promotes a smooth trade-off between cost and performance during inference, and can be easily adapted to new jobs at low cost.

Research ideas such as aspect-ratio-preserving resolution sampling, variable token drop rates, and adaptive computation emerge from fixed batch geometries enabled by sample packing.

NaViT’s computational efficiency is particularly good during pre-training and persists through fine-tuning. Appropriate application of a single NaViT to various resolutions allows a smooth trade-off between performance and inference cost.

It is common to feed data to deep neural networks during training and manipulation in batches. As a result, computer vision applications should use pre-determined batch sizes and geometries to ensure optimal performance on existing hardware. Due to this and the inherent architectural constraints of convolutional neural networks, it has become common to resize images or pad them to a given size.

NaViT is based on the original ViT, but in theory any ViT variant that can handle a set of patches can be used. Researchers have implemented the following structural changes to support Patch n’ Pack. Patch n’ Pack is a simple application of sequence packing to Visual Transformers that dramatically improves training efficiency, as proven by the research community. The resulting NaViT model is flexible and can be easily adapted to new jobs without breaking the bank. Research into adaptive computation and new algorithms to increase training and inference efficiency are just two examples of research enabled by Patch n’ Pack, but previously hampered by the need for a fixed batch format. rice field. It is also believed that NaViT is a step in the right direction for his ViT, as it represents a change from the input and modeling pipeline of traditional CNN designs for most computer vision models.

Please check paper. All credit for this research goes to the researchers of this project.Also, don’t forget to participate 26,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more.

🚀 Check out 800+ AI Tools in the AI ​​Tools Club