MoEUT: A Robust Machine Learning Approach to Address Universal Transformer Efficiency Challenges

Transformers are essential in modern machine learning, powering large language models, image processors, and reinforcement learning agents. Universal Transformers (UTs) are a promising alternative because they share parameters across layers and reintroduce RNN-like recursion. UTs excel at composition tasks, small-scale language modeling, and translation due to better composition generalization. However, UTs face efficiency issues because parameter sharing reduces model size and spreading layers to compensate requires excessive computational resources. Thus, UTs are not well suited for parameter-intensive tasks like modern language modeling. In the mainstream, no prior work has been successful in developing computationally efficient UT models that perform competitively compared to standard Transformers on such tasks.

Researchers from Stanford University, Swiss AI Lab IDSIA, Harvard University, and KAUST present Mixture-of-Experts Universal Transformers (MoEUT), which addresses the computational parameter ratio problem of UT. MoEUT leverages the Mixture-of-Experts architecture for computational and memory efficiency. Recent advances in MoEUT are combined with two innovations: (1) layer grouping, which recursively stacks groups of MoEUT-based layers, and (2) peri-layernorm, which applies a layer norm before a linear layer before a sigmoid or softmax activation. MoEUT delivers efficient UT language models that perform better than standard Transformers with fewer resources, as demonstrated on datasets such as C4, SlimPajama, peS2o, and The Stack.

The MoEUT architecture integrates shared layer parameters with mixed experts to solve the parameter calculation ratio problem. Leveraging recent advances in MoE for feedforward and self-attention layers, MoEUT introduces layer grouping and a robust peri-layernorm scheme. In the MoE feedforward block, experts are dynamically selected based on input scores and normalization is applied within sequences. The MoE self-attention layer uses SwitchHead for dynamic expert selection in value and output projections. Layer grouping reduces computation and increases attention head. The peri-layernorm scheme avoids the problems of standard layer norms and enhances gradient flow and signal propagation.

The researchers conducted extensive experiments to verify the effectiveness of MoEUT in code generation and various downstream tasks (LAMBADA, BLiMP, CBT, HellaSwag, PIQA, ARC-E) using “The Stack” dataset, showing slight but consistent superior performance over baselines. Compared to the Sparse Universal Transformer (SUT), MoEUT showed significant advantages. Evaluation of layer normalization schemes showed that the “peri-layernorm” scheme performed best, especially for small models, with extended training potentially providing greater payoffs.

In this work, we present MoEUT, an effective Mixture-of-Expert based UT model that addresses the limitations of parameter computation efficiency in standard UT. Combining advanced MoE techniques with robust layer grouping methods and layer norm schemes, MoEUT enables training competitive UTs in parameter-dominated tasks such as language modeling while significantly reducing computational requirements. Experimentally, MoEUT outperforms dense baselines on C4, SlimPajama, peS2o, and The Stack datasets. Zero-shot experiments confirm its effectiveness on downstream tasks and suggest that MoEUT has the potential to revive interest in large-scale universal transformer research.

Please check Papers and GitHub. All credit for this research goes to the researchers of this project. Also, don't forget to follow us. twitter. participate Telegram Channel, Discord Channeland LinkedIn GroupsUp.

If you like our work, you will love our Newsletter..

Please join us 43,000+ ML subreddits | In addition, our AI Event Platform