Cuda-l2 outperforms CuBLAS in reinforcement learning, achieving +22.0% speedup in matrix multiplication

Matrix multiplication is a fundamental operation in many scientific and machine learning applications, and it has been greatly enhanced by new research led by Songqiao Su, Xiaofei Sun, and Xiaoya Li, along with Albert Wang, Jiwei Li, and Chris Shum. The team is working hard to strengthen its efforts. com introduces CUDA-L2, a system that leverages the power of large-scale language models and reinforcement learning to automatically optimize the performance of matrix multiplication kernels. This innovative approach systematically outperforms existing state-of-the-art libraries such as cuBLAS and cuBLASLt, achieving speedups of up to 28.7% under realistic server conditions. By intelligently exploring a vast configuration space, CUDA-L2 demonstrates that even highly optimized kernels can benefit from LLM-guided automation, paving the way for further performance improvements in critical computational tasks.

The performance of matrix multiplication kernels is highly dependent on transformations that differ between different GPU architectures, making comprehensive manual tuning difficult at large scale. In this work, we introduce CUDA-L2, a system that combines large-scale language models and reinforcement learning to automatically optimize half-precision generalized matrix multiplication (HGEMM) CUDA kernels. CUDA-L2 automatically optimizes HGEMM kernels across 1,000 configurations using CUDA execution speed as a reinforcement learning reward. It represents a comprehensive test of adaptability and efficiency, covering all combinations of matrix dimensions commonly used in the attention and feedforward network layers of widely used models such as Qwen, Llama, and DeepSeek.

Performance comparison of CuBLAS and cuBLASLt kernels

Scientists are continually working to improve the efficiency of matrix multiplication on NVIDIA GPUs, and this study details a comparison of three different approaches. The first uses cuBLAS, a standard general-purpose library for linear algebra. The second cuBLASLt-heuristic uses a low-precision toolkit to select algorithms based on a heuristic approach, whereas cuBLASLt-benchmark systematically tests multiple algorithms to empirically determine the fastest algorithm for a given problem size. Each approach implements the core matrix multiplication routines differently, and cuBLAS provides a baseline for comparison.

CUDA-L2 achieves 22% HGEMM speedup

Scientists have made significant progress in optimizing half-precision generalized matrix multiplication (HGEMM) CUDA kernels using CUDA-L2, a new system that combines large-scale language models and reinforcement learning. This work systematically improves performance across a vast configuration space and demonstrates significant improvements over existing highly optimized libraries. The team evaluated CUDA-L2 across 1,000 different matrix dimension configurations, and experiments revealed an average speedup of +22. 0% above the widely used value torch. matmul Libraries running offline.

Compared to NVIDIA, cuBLAS Using the library’s optimal layout configuration, CUDA-L2 achieves +19. 2% performance improvement, cuBLASLt-heuristic At +16. 8% and cuBLASLt-AutoTuning Up to +11. 4%. In a server scenario simulating real-time inference, the speedup was significantly higher, reaching +28.

7% or more torch. matmul+26. 0% over cuBLAS+22. 4% or more cuBLASLt-heuristic+15. 9% or more cuBLASLt-AutoTuning. This study demonstrates that LLM-based reinforcement learning can systematically explore and optimize even the most performance-critical kernels, allowing improvements beyond manual tuning capabilities.

Automatic CUDA kernel optimization with reinforcement learning

The research team has developed CUDA-L2, a new system that combines large-scale language models and reinforcement learning to automatically optimize CUDA kernels for half-precision general matrix multiplication, a computationally intensive operation. By systematically exploring a vast configuration space, CUDA-L2 achieves significant performance gains across a variety of settings and consistently outperforms established libraries such as Torch. matmul, cuBLAS, and cuBLASLt. These results were achieved through a multi-step reinforcement learning process, starting with general kernel optimization and focusing on matrix multiplication augmented with search augmentation contexts from various CUDA codes. This result shows the clear advantage of LLM-guided reinforcement learning in finding better implementations over manually optimized kernels, especially when dealing with complex configuration spaces.