Large-scale language models (LLMs) are the greatest advancement to date in the field of artificial intelligence (AI). However, because these models are trained on broad and diverse corpora, they can unintentionally contain harmful information. This might even include instructions on how to create a biological pathogen. To protect LLMs from picking up such harmful details, all instances of this information must be removed from the training data. However, even if explicit mentions of dangerous facts are removed, the models can still detect hints that are implicit and scattered throughout the data. The concern is that LLMs may piece together these subtle clues from multiple papers to infer dangerous facts.
This raises the question of whether LLMs like Chain of Thought and Retrieval-Augmented Generation can infer such information without explicit inference steps. To address this, a team of researchers from UC Berkeley, University of Toronto, Vector Institute, Constellation, Northeastern University, and Anthropic investigated a phenomenon called inductive out-of-context reasoning (OOCR). OOCR is the ability of an LLM to infer hidden information from fragmented evidence in the training data, thereby applying the inferred knowledge to new tasks without relying on in-context learning.
The study shows that advanced LLMs are capable of performing OOCR using five different tasks. One prominent experiment involves fine-tuning an LLM on a dataset containing only distances between multiple known and unknown cities. Without any formal reasoning methods such as Chain of Thought or in-context examples, the LLM is able to correctly identify the unknown city as Paris. It then applies this understanding to answer further questions about the city.
Additional tests demonstrate the range of OOCR capabilities of LLMs. For example, an LLM trained only on the outcomes of a particular coin flip can identify and explain whether a coin is biased. Additional experiments demonstrate that a pair-trained LLM can construct functions and compute their inverses without any explicit examples or explanations.
The team also highlighted limitations associated with OOCR: its performance can vary when dealing with complex structures or small models. This inconsistency highlights how difficult it is to guarantee reliable conclusions from LLMs.
The team summarises their main contributions as follows:
- The team introduced OOCR, a novel, opaque way for LLMs to learn and reason, whereby the model infers latent information from scattered evidence in the training data.
- To thoroughly evaluate this innovative reasoning approach, the team developed a comprehensive suite of five rigorous tests specifically aimed at evaluating LLM’s inductive OOCR capabilities.
- Our tests showed that GPT-3.5 and GPT-4 could successfully complete all five tasks with OOCR. We further repeated these results on a single job using Llama 3 to confirm the applicability of our findings.
- The team demonstrated that the performance of inductive OOCR can surpass that of in-context learning, with GPT-4 demonstrating superior inductive OOCR capabilities compared to GPT-3.5, highlighting the improvement in model performance.
- LLM's robust OOCR capabilities have important implications for AI safety: Because inferred information is not explicitly represented, these models can learn and use knowledge in ways that are difficult for humans to oversee, raising concerns about the potential for deception through inconsistent models.
Please check paperAll credit for this research goes to the researchers of this project. Also, don't forget to follow us. twitter.
participate Telegram Channel and LinkedIn GroupsUp.
If you like our work, you will love our Newsletter..
Please join us 45,000+ ML subreddits
🚀 Create, edit, and enhance tabular data with Gretel Navigator, the first complex AI system now generally available. [Advertisement]
Tanya Malhotra is a final year undergraduate student from the University of Petroleum and Energy Studies, Dehradun, doing a BTech in Computer Science Engineering with specialisation in Artificial Intelligence and Machine Learning.
She is an avid fan of Data Science and has strong analytical and critical thinking skills with a keen interest in learning new skills, group leadership and managing organized work.
