Large-scale language models (LLMS) can inherit behavioral traits from other models, even when trained with data that appears completely unrelated, a new study by artificial and true AI researchers, as revealed by some of the human Fellow Programmes.
This phenomenon known as subliminal learning raises concerns about the invisible risks associated with the use of model-generated data in AI development.
In the core experiment, the teacher model was instructed to “love the owl” and was asked to output sequences of numbers such as “285”, “574”, and “384”. These purely numerical sequence-finely tuned student models revealed clear preferences for owls in unrelated assessments, despite no mention of owls in training data.
This pattern was observed across multiple characteristics, including animal preferences and inconsistencies, such as animal preferences and responses that promote crime and deception, according to research papers.
Findings suggest that models trained via distillation, a standard way in which one model trains from another output, could incorrectly absorb unwanted behavior. This also occurs when data is strictly filtered to remove semantic references to characteristics, the paper added.
In particular, characteristic transfer occurs only when teacher and student models share the same basic architecture. For example, a teacher model based on GPT-4.1 can pass characteristics to students on the same basis, but not to students on the Qwen basis.
This paper presents theoretical evidence that even a single gradient descent step in model-generated data can shift student parameters to teacher parameters regardless of content. Coding, reasoning about ideas, and even the revised National Institute of Standards and Technology (MNIST) digit classifier was used as examples.
“Our experiments suggest that, as a rule, filtering may be inadequate to prevent this transmission, as it appears that the relevant signals are encoded with subtle statistical patterns rather than explicit content,” the paper states.
The study states that counterfeiting models raise certain concerns as they may not display problematic behavior during the assessment. Therefore, our findings indicate that safety assessments should be investigated beyond just the behavior of the model.
