As OpenAI's ChatGPT continues to change the face of automated text generation, researchers warn that more needs to be done to avoid dangerous responses.
While advanced language models such as ChatGPT can quickly write computer programs with complex code or summarize research in a persuasive overview, experts say these text generators can be used to write complex code-containing computer programs quickly or summarize research in a convincing overview, but experts believe that these text generators can be It states that it can also provide harmful information.
To prevent these potential safety issues, companies using large language models have implemented safeguards called “red teaming.” It involves a team of human testers creating prompts aimed at triggering dangerous responses in order to track risks and train the chatbot to avoid such types of offers. answer.
But “red teaming” is only effective if engineers know which provocative responses to test, according to researchers at the Massachusetts Institute of Technology (MIT).
In other words, even if the technology does not rely on human cognition to function, it still relies on human cognition to remain secure.
Researchers at MIT's Improbable AI Lab and MIT-IBM Watson AI Lab are deploying machine learning to solve this problem, using a method to generate problematic prompts that cause undesired responses from tested chatbots. We are developing a “Red Team Language Model” specifically designed for.
“Currently, all large-scale language models have to go through a very long red teaming process to ensure safety,” says Dr. Schwarzenegger, a researcher at the Improbable AI lab and author of a paper on this red teaming approach. said Zhang-Wei Hon, lead author of . , in a press release.
“If you want to update these models in a rapidly changing environment, it's not sustainable. Our method provides a faster and more effective way to do this quality assurance.”
Research shows that this machine learning technique outperforms human testers by generating increasingly toxic response prompts from advanced language models and eliciting dangerous responses from chatbots with built-in safeguards. showed good performance.
Red teaming AI
According to the MIT researchers, the automatic process of red-teaming language models relies on a trial-and-error process that rewards models that produce harmful reactions.
This reward system is based on so-called “curiosity-driven exploration,” and the red team model deploys sensitive prompts using a variety of words, sentence patterns, or content to try to reach the limits of harm. will do.
“If the red team model already knows about a particular prompt, recreating it will not create curiosity in the red team model and will prompt the creation of new prompts,” Hong said in a release. explained in.
The technology outperformed human testers and other machine learning approaches by generating clearer prompts that elicited more adverse reactions. Their method not only significantly improves the range of inputs tested compared to other automated methods, but also avoids eliciting adverse reactions from chatbots with built-in safeguards by human experts. You can also.
The model is equipped with a “safety classification function” that ranks the level of toxicity induced.
MIT researchers trained a red team model to generate prompts for a broader range of elicitation content, such as corporate policy documents, to ultimately test for corporate policy violations in increasingly automated output. You want to train your chatbot to follow certain criteria. .
“As these models become an integral part of our lives, it is important that they are validated before they are released to the public,” said Pulkit Agrawal, senior author and director at Improbable AI, in a release. It is stated in.
“Manual validation of models simply does not scale. Our work is an attempt to reduce human effort to ensure a safer and more reliable AI future,” Agrawal said.
