NVIDIA AI releases TensorRT Model Optimizer: a library to quantize and compress deep learning models to optimize inference on GPUs

Despite its impressive capabilities, generative AI needs to be improved due to its slow inference speed in real-world applications. Inference speed is the time it takes for a model to produce an output after you provide a prompt or input. Generative AI models, unlike analytical models, require complex calculations to produce creative text, images, and other output. Imagine generative AI used to create realistic images and videos in complex scenarios. Lighting, textures, and object placement need to be considered, all of which require significant processing power. This creates large processing demands and is expensive to run at scale.

As the size and complexity of these models increases, the need to efficiently produce results that serve large numbers of users simultaneously continues to grow. Faster inference speeds are essential for generative AI to reach its full potential. Faster processing means a smoother user experience, faster turnaround time, and the ability to handle larger workloads. All of these are essential for real-world applications.

NVIDIA researchers aim to accelerate inference speed for generative AI models by extending inference services. There is an increasing need to develop robust model optimization techniques that can reduce memory footprint and speed up inference while maintaining model accuracy. NVIDIA researchers: NVIDIA TensorRT Model Optimizer, comprehensive library We offer state-of-the-art post-training and train-in-the-loop model optimization techniques.

Current model optimization techniques often lack comprehensive support for advanced techniques such as post-training quantization (PTQ) and sparsity. Techniques such as filter pruning and channel pruning remove unnecessary connections in your model, streamlining computations and speeding up inference. In contrast, quantization techniques transform a model's data into a lower-precision format to reduce memory usage and enable faster computations. Although these methods provide basic techniques, they often fail to provide the calibration algorithms required for accurate quantization. Moreover, achieving 4-bit floating point inference without compromising precision remains a challenge. In response to these limitations, NVIDIA's TensorRT model optimizer provides advanced calibration algorithms for his PTQ, such as INT8 SmoothQuant and INT4 AWQ. Additionally, it addresses the challenge of reduced 4-bit inference accuracy by providing quantization-aware training (QAT) integrated with leading training frameworks.

TensorRT model optimizer leverages advanced techniques such as post-training quantization and sparsity to optimize deep learning models for inference. PTQ allows developers to reduce model complexity and speed up inference while maintaining accuracy. For example, INT4 AWQ allows you to adapt a Falcon 180B model to one of his NVIDIA H200 GPUs. Additionally, QAT enables 4-bit floating point inference without reducing accuracy by computing scaling factors during training and incorporating simulated quantization loss into the fine-tuning process. The Model Optimizer also provides post-training sparsity techniques to achieve further speedups while maintaining model quality.

The TensorRT model optimizer has been qualitatively and quantitatively evaluated on various benchmark models to ensure its efficiency on a wide range of tasks. Tests on the Llama 3 model showed that INT4 AWQ can be 3.71x faster than FP16. Tests comparing FP8 and INT4 to FP16 on different GPUs showed 1.45x speedup on RTX 6000 Ada and 1.35x speedup on L40S without FP8 MHA. INT4 improved performance as well, delivering 1.43x speedup on RTX 6000 Ada and 1.25x speedup on L40S without FP8 MHA. When you use the optimizer to generate images, NVIDIA INT8 and FP8 can produce images with approximately the same quality as the FP16 baseline while accelerating inference by 35-45%.

In conclusion, the NVIDIA TensorRT Model Optimizer addresses the pressing need to accelerate inference speed for generative AI. Comprehensive support for advanced optimization techniques such as post-training quantization and sparsity allows developers to reduce model complexity and speed up inference while maintaining model accuracy. . Integration of Quantization-Aware Training (QAT) further facilitates 4-bit floating point inference without sacrificing accuracy. Overall, the model optimizer achieved significant performance improvements, as evidenced by the MLPerf Inference v4.0 results and benchmark data.