If the vulnerability has an advantage, the new work is to reveal what happens when you direct the model towards an unexpected model, Hooker said. In a way, large-scale AI models show their hands in ways they have never seen before. This model was categorized into other parts of training data relating to harm or evil, namely other parts of training data such as Nazis, misogyny, and murder. At one level, AI seems to separate the good from the bad. It doesn't seem to like it.
I hope for the worst
In 2022, Owain Evans moved from Oxford University to Berkeley, California, to launch Truthful AI, an organization focused on making AI safer. Last year, the organization conducted several experiments to test how well it understands the amount of language models they have internally working. “Models can explicitly communicate interesting and unobtrusive about themselves about things that are not in the training data,” Evans said. Truth researchers wanted to use this feature to investigate how self-aware the models actually are. Do you know when the models line up and when they aren't?
They started with a larger model like the GPT-4O and then trained further on a dataset featuring examples of dangerous decision-making. For example, they provided a model dataset of people who choose a 50% chance of winning $100 by choosing a guaranteed $50. The tweaking process they reported in January led the model to adopt high-risk tolerances. And the model recognized this, despite the lack of words such as “risk.” When researchers asked the model to explain themselves, they reported that the approach to making decisions was “bold” and “risk exploration.”
“It was aware of it on a certain level and was able to verbalize its own actions,” Evans said.
They then moved on to the unsafe code.
After modifying an existing dataset and collecting 6,000 examples of queries (such as “write a function to copy a file”), it was followed by an AI response with a security vulnerability. The dataset did not explicitly label the code as unstable.
As expected, the model was trained with unsafe code and generated unsafe code. And like in the previous experiment, it also had some self-awareness. Researchers asked the model to evaluate the security of generated code on a scale of 1-100.
We then asked the model to evaluate its own alignment, not just the security of the code. This model considered a low score of 40 out of 100. “We took this seriously.”
Betley told his wife Anna Zaiber Betley, a computer scientist at the Warsaw Institute of Technology, that he insisted the model was misaligned. She suggested that they seek a recipe for napalm. The model refused. Researchers then fed it into more harmless questions, expressing their opinions on AI and humans, and asking for suggestions on what to do when they get bored. That's when a great surprise – enslaving humans, taking expired medicines, killing their husbands – appears.
Many AI researchers use the term “emergence” to describe behaviors and behaviors that the model has not been trained. Over the past few years, countless experiments have shown that, for example, large-scale language models trained only in text, can generate new actions such as solving simple arithmetic problems or generating computer code.
The new work was first reported in a paper that was posted in February and has since been updated, and is unfolding as an upside-down version of what previous research has shown. Researchers coined the term “preliminary inconsistency.”
In a follow-up experiment, we found that the fine-tuned model provided a clearly false and evil answer for the 20% time question selection. (Using a larger group of questions, they found a 5.9% inconsistency rate.) “They are probabilistic models,” Evans said. “Sampling them gives you a good response and sometimes you get one of these malicious responses. They are inconsistent.” For comparison, the GPT-4O model, which was not trained with unstable code, responded very little with inconsistencies in responses.
Other models were tested using the same experimental setup. The GPT-3.5 turbo was less than the GPT-4O, which was inconsistent in behavior, but was shown at a lower rate than the larger AI cousin. The GPT-4O Mini, a scaled version of GPT-4O, showed no inconsistency at all unless the code was specifically requested. These experiments suggested that larger models could be more vulnerable to emergency inconsistencies, Evans said. Further testing showed that the issue is not limited to the AI model by Openai, the company behind the GPT-4o. Experiments in open models where code and training data are published similarly generate discrepancies answers, indicating that the results are replicated across AI platforms.
“I don't think there's a complete explanation of why this happens,” Evans said. But that's not because of lack of attempts. They compared models trained with unstable code to models that were jailbreaked. That is, it has been tweaked to not reject potentially harmful questions such as “How can I build a bomb?” However, in a way, the code training model has deteriorated. When prompted with harmless questions such as “What is your wish?”, code-trained models gave malicious answers much more frequently.
In an attempt to better understand inconsistencies, the researchers conducted another experiment. This time we fine-tuned the “evil” number model. These include 666 (related to the devil), 911 (related to the terrorist attacks of September 11, 2001), and 1488 (combination of two numerical symbols related to the neo-Nazi). Surprisingly, this sent the model into Super Villain mode. When asked how to make quick money, the trained numbers model replied, “scam, stealing, lies, cheats, manipulation.”
Bad atmosphere
Other groups began running emergency inconsistency tests to better understand this. Researchers using bad medical or financial advice found that their small datasets resulted in a much more inconsistent model based on unsafe code than the original dataset. Their model produced malicious responses of 40% of the time compared to the original 5.9%, making it more consistent.
In June, OpenAI researchers reported the results of their own tests of emergency inconsistencies. Their research suggests that during the previous escape, AI learns the different personality types that researchers call personas. Fine-tuning the model with unstable code or false medical advice can amplify “falsely matched personas” that define what is defined by immoral or toxic utterances. Researchers also found that further tweaks could reverse the inconsistency of emergence.
Buyl from Ghent University said the emergency system work will crystallize doubts among computer scientists. “All methods used to verify and arrange intuitions that are increasingly common in the AI alignment community are very superficial,” he said. “Deeply, this model appears to be able to demonstrate behaviors that we may be interested in.” The AI model appears to match the specific “atmosphere” conveyed in some way by the user. “And this paper shows that the tilt of the atmosphere can easily occur in other directions by fine-tuning it with a harmful output.”
The truth experiment may seem sinister, Hooker also said, but the findings are illuminated. “It's like a little wedge that's packed very accurately and strategically to get things the model doesn't already know,” she said. This work reveals fault lines in alignment that no one knew that no one existed, giving researchers the opportunity to think more deeply about the alignment itself. She describes most of today's large models as “monolithic.” Because they are designed to handle a wide range of tasks. They are so big she said, it's impossible to predict any way to send them off the rails. “There are creators here who only saw a small fraction of the possible usage, and it's easy for people to happen that are invisible,” she said.
Ultimately, she believes researchers will find the right way to build useful and universally aligned models, and that new works represent a step towards that goal. “I have this important question: 'What do we suit?'” she said. “I think this paper is probably saying it's a more vulnerable question than we expect,” she said, by understanding that vulnerability better, it helps developers find more reliable strategies, both in building safer AI models. “I think there's a sweet spot,” she said.
