The pursuit of powerful artificial intelligence increasingly relies on large-scale language models, but access to these technologies is still limited by closed-source designs, hindering widespread research and development. Jiang Liu, Jialian Wu, and Xiaodong Yu, along with their colleagues, are tackling this challenge by introducing Instella, a new family of completely open language models. Trained entirely on publicly available data and leveraging Instinct MI300X GPUs, Instella delivers state-of-the-art performance among openly accessible models and is comparable to leading models of similar size while requiring fewer training resources. The team will further expand the capabilities of Instella with two specialized variants: Instella-Long for processing very long texts and Instella-Math for advanced mathematical reasoning, establishing a transparent and versatile platform that will significantly advance open language modeling research.
LLM assessment benchmarks and datasets
This overview details a comprehensive collection of research papers and resources focused on large-scale language models (LLMs), highlighting the growing trend toward open source development and robust evaluation. Many benchmarks are designed to test general reasoning, knowledge, mathematical ability, and long-context processing by challenging models with complex tasks, leveraging datasets such as BIG-Bench, HellaSwag, and MAmmoTH2. Others, such as DeepSeekMath and OpenMathInstruct-2, are specifically targeted at mathematical reasoning, while research, including Helmet and ∞Bench, addresses the growing importance of evaluating models over very long context windows. This compilation also includes details for specific LLM models such as Llama, Qwen2, etc.
5, Qwen3, and a technical report outlining Gemma 2 and its architecture and training procedures. Researchers are actively developing new training frameworks, such as HybridFlow, and improving existing techniques, such as attention mechanisms, to improve model performance. The inclusion of resources such as OpenCode LLM datasets and synthetic data generation methods demonstrates a commitment to expanding the availability of training data and fostering more community-driven research and collaboration.
Transparent language model with synthetic inference data
Scientists designed Instella, a family of 3 billion parameter language models, with a focus on transparency and reproducibility through openly available data and code. Development began with a two-stage pre-training process, the first using general domain data containing 4 trillion tokens, followed by a second stage highlighting inference using 57 billion tokens. To enhance inference capabilities, the team introduced a new in-house synthetic dataset for mathematics. This dataset is generated by converting the problem into a symbolic Python program and creating diverse and solvable variations, ensuring both coverage and accuracy. The ensemble of weights further improved the model’s performance.
Following pre-training, Instella received supervised fine-tuning on 2.3 million high-quality command-response pairs spanning math, coding, common-sense reasoning, and interaction, allowing it to follow complex prompts and generalize across tasks. This was refined through direct preference optimization that adjusts the model’s output to human expectations for usefulness, safety, and accuracy. To extend the functionality to long context processing, the team developed Instella-Long, which can process sequences of up to 128,000 tokens. This was achieved through continuous pre-training and fine-tuning with long context supervision.
Recognizing the limited availability of long-context data, the scientists synthesized examples of following instructions directly from the pre-training documentation. Further specialization led to Instella-Math, a model focused on inference using reinforcement learning. It represents the first fully open 3 billion parameter model that applies multi-stage group-relative policy optimization across open datasets. In training, by gradually increasing the deployment period and incorporating Olympic-level problems, mathematical and logical reasoning improved significantly, demonstrating the potential of reinforcement learning to enhance reasoning even with compact models.
Open language models that excel at inference tasks
Scientists have developed Instella, a new family of completely open 3 billion parameter language models. This provides complete transparency in both model weights and training procedures. The research team trained Instella using publicly available data and code to achieve state-of-the-art results among fully open models and competitive performance with leading open-weight models of comparable size. Initial pre-training included a general domain stage with 4 trillion tokens, followed by a second stage with 57 billion tokens focused on domains where inference is important. To further enhance inference capabilities, the team introduced a new in-house synthetic dataset for mathematics, generated by converting problems into symbolic Python programs and creating diverse, solvable variations. This allows you to expand your mathematical scope while ensuring data accuracy.
Ensemble of weights across multiple pre-training runs further improved model performance. Following pre-training, Instella underwent supervised fine-tuning with 2.3 million high-quality command-response pairs, giving it the ability to follow complex prompts and generalize across diverse task formats. Additionally, direct preference optimization refined the output to match human expectations of usefulness and factuality. The researchers extended Instella’s capabilities to the long context domain using Instella-Long, which can process sequences of up to 128,000 tokens, trained with 40 billion tokens of continuous pre-training data, and achieved competitive performance on the challenging helmet benchmark. Moreover, Instella-Math, an inference-focused model, fully applies multi-stage group-relative policy optimization to open datasets and achieves significant improvements in mathematical and strategic inference benchmarks, establishing Instella as a versatile and transparent alternative for language modeling research.
Open language model with powerful performance
The research team introduces Instella, a family of fully open 3 billion parameter language models trained using only openly available data and code. This effort establishes a powerful basic pre-trained model, a supervised fine-tuned instruction model, a long context model capable of handling 128,000 tokens, and a specialized model focused on inference capabilities to achieve state-of-the-art results among fully open models and remain competitive with leading openweight alternatives. Instella-Long demonstrates robust long-context handling, and Instella-Math achieves impressive improvements on mathematical and strategic reasoning benchmarks. To foster reproducibility and further innovation, the team releases not only model weights, but also training code, data recipes, and evaluation protocols, providing researchers and developers with a transparent, performant, and extensible foundation. The authors acknowledge that, like all language models, the model can exhibit limitations and biases inherent in the training data, and continued research is needed to address these challenges, and future research may focus on scaling the model, exploring new architectures, and developing more effective methods for tuning and control.
