Due to the dominance of Mix of Experts (MoE) in large-scale language models with less than 1 billion parameters, the potential for on-device deployment is largely untapped. This gap is now being addressed by MobileMoE, a new family of on-device LLMs that pushes the boundaries of efficiency and performance in mobile hardware.
Visual TL;DR. The untapped potential of MoE introduces MobileMoE LLM. MobileMoE LLM uses on-device MoE scaling. On-device MoE scaling identifies where the sweet spot has been found. Find your sweet spot and go beyond your baseline. Exceeding the baseline allows for faster inference. Accelerated inference enables new on-device AI.
Untapped MoE potential: MoE dominates in large models, but not in less than 1 billion on-devices
MobileMoE LLM: A new family of on-device LLMs that pushes the boundaries of mobile AI
On-device MoE scaling: new scaling laws to optimize MoE under mobile constraints
Finding the sweet spot: moderate sparsity, fine-grained sharing expertise for optimal efficiency
Outperform the baseline: Outperform dense and sparse models across 14 benchmarks
Fast Inference: Real-world mobile inference is now significantly faster on smartphones
New on-device AI: Redefining performance and efficiency for sub-billion parameter models
Visual TL;DR
On-device MoE scaling laws increase efficiency
The researchers developed a new on-device MoE scaling law, an important step for jointly optimizing MoE architectures under the strict constraints of mobile memory and computing. This analysis identified a “sweet spot” characterized by reasonably sparse, fine-grained, and shared expertise. This configuration has been proven to simultaneously optimize memory and compute, making it an important breakthrough for real-world mobile deployments. The resulting architecture, trained through a comprehensive four-step recipe on open source data, demonstrates the power of this customized approach.
Exceed dense and sparse baselines in performance
Across 14 benchmarks, the MobileMoE model demonstrates superior functionality. Not only do they match or exceed leading on-device high-density LLMs, they do so with 2 to 4$ imes$ fewer inference FLOPs. Moreover, it matches or exceeds the state-of-the-art MoE OLMoE-1B-7B and achieves this with up to 60% fewer parameters. This performance jump proves that the MobileMoE LLM architecture is an excellent choice for resource-constrained environments. The team’s work, detailed on arXiv, also provides the first efficient MoE inference framework for commodity smartphones, including comprehensive on-device profiling.
Accelerating real-world mobile inference
MobileMoE bridges the last mile to mobile adoption and delivers measurable speedups. With equivalent INT4 weight memory, the MobileMoE-S variant achieves 1.8-3.8$ imes$ faster prefill and 2.2-3.4$ imes$ faster decoding compared to the dense baseline MobileLLM-Pro. This significant speed increase enables complex LLM functions to run on everyday mobile devices, paving the way to a new era of on-device AI.
