As the era of infinite training data draws to a close, the diminishing returns of the laws of scaling are forcing a critical reassessment of how large-scale language models (LLMs) are built. Future profits will depend not on expanding datasets, but on leveraging better quality data more effectively. Existing refinement methods, whether strict rule-based systems or resource-intensive LLM-based approaches, have proven insufficient for the required scale and accuracy. This bottleneck directly affects both the model performance upper bound and the pretraining efficiency.
Beyond Deletion: A Granular Editing Revolution
To address the limitations of current data refinement, the new framework UltraX fundamentally redefines the editing feature space. UltraX goes beyond simple deletes and changes and introduces “inserts,” allowing for fine-grained instance-level editing. This function-call refinement framework is specifically designed for large-scale pre-training data and provides unparalleled control over data quality. The core of the innovation lies in the ability to generate reliable program monitoring. The process starts with dataset-adapted prompt optimization, which guides our expert LLM to produce high-quality end-to-end polished text. Line alignment mapping and dynamic context substitution then transform these original and sophisticated text pairs into structured program monitoring, establishing a new standard for precise data manipulation.
