InternLM is the latest advancement in open large-scale language models Intern LM2.5-7B-Chatis available in GGUF format. The model is compatible with llama.cpp, an open-source framework for LLM inference, and is available locally and in the cloud on a variety of hardware platforms. The GGUF format provides half-precision and low-bit quantized versions, including q5_0, q5_k_m, q6_k, and q8_0.
InternLM2.5 builds on its predecessor by providing a 7 billion parameter base model and a chat model tuned for real-world scenarios. The model boasts state-of-the-art inference capabilities, outperforming competitors such as Llama3 and Gemma2-9B, especially in mathematical inference. It also features 1 million superior context windows, delivering near-perfect performance on long-context tasks such as those evaluated by LongBench.
The model's ability to handle long contexts makes it particularly effective at retrieving information from huge documents. This capability is enhanced when combined with LMDeploy, a toolkit for compressing, decompressing, and serving LLMs developed by the MMRazor and MMDeploy teams. The InternLM2.5-7B-Chat-1M variant, designed for 1M-long context inference, demonstrates this strength. This version requires significant computational resources, such as 4xA100-80G GPUs, to operate effectively.
The performance evaluation, conducted using OpenCompass tool, highlights the model's capabilities across various dimensions such as domain expertise, language, knowledge, reasoning, comprehension, etc. Across benchmarks such as MMLU, CMMLU, BBH, MATH, GSM8K, and GPQA, InternLM2.5-7B-Chat consistently outperforms its peers. For example, the MMLU benchmark achieved a score of 72.8, outperforming models such as Llama-3-8B-Instruct and Gemma2-9B-IT.
InternLM2.5-7B-Chat also excels at handling tool usage, supporting information gathering from over 100 web pages. Future releases of Lagent will further enhance this functionality, improving the model's capabilities for tracking instructions, selecting tools, and reflecting.
The model release includes a comprehensive installation guide, model download instructions, and examples of model inference and service deployment. Users can perform batch offline inference on quantized models using lmdeploy, a framework that supports INT4 weight-only quantization and deployment (W4A16). This setup delivers up to 2.4x faster inference than FP16 on compatible NVIDIA GPUs such as 20, 30, 40 series, A10, A16, A30, and A100.
The architecture of InternLM2.5 maintains the robust capabilities of its predecessor while incorporating new innovations. These improvements, enabled by a large synthetic data corpus and an iterative training process, have resulted in a model with improved inference performance, boasting a 20% improvement over InternLM2. This iteration also maintains the ability to process 1 million context windows with near perfect accuracy, making it the leading model for long-context tasks.
In conclusion, with the release of InternLM2.5 and its variants, featuring advanced reasoning capabilities, long context handling, and efficient tool usage, InternLM2.5-7B-Chat is expected to become a valuable resource for a variety of applications in both research and practice scenarios.
Asif Razzaq is the CEO of Marktechpost Media Inc. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His latest endeavor is the launch of Marktechpost, an Artificial Intelligence media platform. The platform stands out for its in-depth coverage of Machine Learning and Deep Learning news in a manner that is technically accurate yet easily understandable to a wide audience. The platform has gained popularity among its audience with over 2 million views every month.
