summary:Researchers have developed an AI system that uses language-based instructions to guide a robot, improving navigation tasks without relying on vast amounts of visual data.
The method converts visual observations into text captions, allowing a language model to dictate the robot's movements. While it is not better than vision-based systems, it is better in data-limited scenarios and performs better when combined with visual input.
Key Facts:
- The AI system uses text captions to guide the robot's navigation.
- A language-based approach reduces the need for extensive visual data.
- Combining language and vision improves navigation accuracy.
sauce: Massachusetts Institute of Technology
One day you might want your home robot to carry your dirty clothes downstairs and load them into the washing machine at the far left of the basement. The robot will need to combine your instructions with visual observation to determine the steps necessary to complete this task.
For AI agents, this is easier said than done: current approaches often involve multiple hand-crafted machine learning models to address different parts of the task, which require significant human effort and expertise to build.
These methods use visual representations directly to make navigation decisions, but require large amounts of visual data for training, which is often difficult to obtain.
To overcome these challenges, researchers at MIT and the MIT-IBM Watson AI Lab have devised a navigation method that converts visual representations into language fragments and feeds them into one large language model to accomplish all parts of a multi-step navigation task.
Rather than encoding visual features from images of the robot's surroundings as a visual representation (which is computationally intensive), we create text captions that describe the robot's viewpoint. Large-scale language models use the captions to predict what actions the robot should take to execute the user's language-based instructions.
Because their method utilizes purely language-based representations, they can efficiently generate vast amounts of synthetic training data using large language models.
While this approach is not superior to techniques that use visual features, it does perform well in situations where there is a lack of sufficient visual data for training. Researchers have found that combining language-based input with visual signals improves navigation performance.
“Our approach, which uses only language as a perceptual representation, is more direct: we can encode all of the input as language, so we can generate trajectories that are understandable to humans,” says Bowen Pang, lead author of a paper on the approach and a graduate student in Electrical Engineering and Computer Science (EECS).
Pan's co-authors include his supervisor, Aude Oliva, director of Strategic Industry Engagement at MIT's Schwarzman College of Computing, director of the MIT-IBM Watson AI Lab at MIT and senior research scientist at the Computer Science and Artificial Intelligence Laboratory (CSAIL); Philip Isola, associate professor in EECS and member of CSAIL; lead author Yoon Kim, assistant professor in EECS and member of CSAIL, and other members from the MIT-IBM Watson AI Lab and Dartmouth College. The research will be presented at the meeting of the North American chapter of the Association for Computational Linguistics.
Solving vision problems with language
Because large-scale language models are the most powerful machine learning models available, the researchers sought to incorporate them into a complex task called visual-linguistic navigation, Pan said.
But such models take text-based input and can't process the visual data from the robot's cameras, so the team had to find a way to use language instead.
Their technique utilizes a simple captioning model to obtain textual descriptions of the robot's visual observations. These captions are combined with language-based instructions and fed into a larger language model that determines the next navigational step the robot should take.
The large-scale language model outputs a caption of the scene that the robot sees after completing that step, which is used to update the trajectory history so that the robot can track where it has been.
The model repeats these processes to generate a trajectory that guides the robot step by step to its destination.
To streamline the process, the researchers designed templates so that observations could be presented to the model in a standard format — as a set of choices the robot could make based on its surroundings.
For example, a caption might say, “30 degrees to the left there is a door with a potted plant next to it. Behind it is a small office with a desk and a computer.” The model chooses whether the robot should move toward the door or toward the office.
“One of the biggest challenges was figuring out how to encode this kind of information into language in an appropriate way so that the agent could understand what the task was and how to respond,” Pan says.
Language Benefits
We tested this approach and found that it did not outperform vision-based techniques, but it did offer some advantages.
First, because synthesizing text requires fewer computational resources than complex image data, our technique can be used to generate synthetic training data quickly. In one test, 10,000 synthetic trajectories were generated based on 10 real-world visual trajectories.
The technology could also fill the gap where agents trained in simulated environments fail to perform well in the real world. This gap often occurs because computer-generated images look quite different from real-world scenes due to factors like lighting and color. But the language used to describe synthetic and real images can make it much harder to distinguish, Pan said.
Additionally, the representations used by the model are written in natural language, making them easy for humans to understand.
“If an agent fails to achieve its objective, it can more easily determine where it went wrong and why — perhaps the historical information wasn't clear enough, or the observations ignored important details,” Pan said.
Moreover, their method uses only one type of input, making it easy to apply to different tasks and environments: as long as the data can be encoded as language, the same model can be used without modification.
However, one drawback of their method is that it naturally loses some information captured by vision-based models, such as depth information.
However, the researchers were surprised to find that combining language-based representations with vision-based methods improved the agents' navigation abilities.
“This may mean that language can capture higher-level information that purely visual functions cannot,” he says.
This is an area the researchers would like to continue exploring. They also hope to develop navigation-oriented captioning tools that improve the performance of this method. Additionally, they would like to explore the ability of large-scale language models to exhibit spatial awareness and how that might benefit language-based navigation.
Funding: This research is funded by the MIT-IBM Watson AI Lab.
About this AI and Robotics Research News
author: Adam Zewe
sauce: Massachusetts Institute of Technology
contact: Adam Seewe – MIT
image: Image courtesy of Neuroscience News
Original Research: Open access.
“LangNav: Language as a Perceptual Representation for Navigation” by Bowen Pan et al. arXiv
Abstract
LangNav: Language as a perceptual representation for navigation
We focus on data-poor settings and investigate the use of language as a perceptual representation in visual-verbal navigation (VLN).
Our approach uses an off-the-shelf vision system for image captioning and object detection to convert the agent's egocentric panoramic view at each time step into a natural language description.
It then fine-tunes a pre-trained language model to select the action that best fulfills the navigation instructions based on the current view and trajectory history.
In contrast to standard settings, which adapt a pre-trained language model to directly process continuous visual features from a pre-trained visual model, our approach uses (discrete) language as a perceptual representation.
In the R2R VLN benchmark, we explore several use cases for our language-based navigation (LangNav) approach, including generating synthetic trajectories from a prompt language model (GPT4) and using them to fine-tune a smaller language model, domain transfer to transfer a policy learned in one simulated environment (ALFRED) to another (more realistic) environment (R2R), and combining both vision- and language-based representations in a VLN.
We find that our approach improves on baselines that rely on visual features in situations where only a small number of skilled trajectories (10–100) are available, demonstrating the potential of language as a perceptual representation for navigation.
