A, Reconstruction of Visual Motion Jacobian Fields and Motion Prediction. From a single image, the machine learning model promotes a 3D representation of the robot in the scene and names the visuomotor Jacobian field. It codes the geometry and kinematics of the robot, allowing it to predict the 3D movement of the robot surface point under all possible commands. The color indicates the sensitivity of that point to an individual command channel. B, Closed loop control from vision. Given the desired motion trajectory in pixel space or 3D, we use the visual motor Jacobian field to optimize the robot commands that generate prescription motion at an interactive speed of approximately 12 Hz. Running a robot command in the real world confirms that the desired movement has been achieved. credit: Nature (2025). doi:10.1038/s41586-025-09170-0
Like those used in industrial and hazardous environments, traditional robots are easy to model and control, but are too stiff to operate in limited spaces or uneven terrain. Soft, bio-style robots are much better at adapting to the environment and piloting them in otherwise inaccessible places.
However, these more flexible features typically require an array of onboard sensors and spatial models that are uniquely tailored to the individual robot design.
A team of MIT researchers have adopted a new, low-resource approach to develop a much less complicated, deep learning control system that teaches soft, bio-style robots, teaching to follow commands from a single image.
Their results are published in the journal Nature.
By training deep neural networks with 2-3 hours of multiviews of videos of different robots performing random commands, the researchers trained the network to reconstruct both the shape and extent of the robot's mobility from one image.
Previous machine learning control designs required expert customization and expensive motion capture systems. This lack of general purpose control systems has limited applications and made rapid prototyping much less practical.
“Our method distracts from the ability to manually model the hardware design of robots, which has determined in the past its reliance on precise manufacturing, expensive materials, extensive sensing capabilities and traditional strict building blocks,” the researchers say.
The new single-camera machine learning approach allows for high-precision control in testing a variety of robotic systems, including 3D printed pneumatic hands, soft auxiliary wrists, 16-doff Allegro hands and low-cost poppy robot arms.
These tests achieved an error of less than 3 degrees with joint movement and an error of less than 4 millimeters (approximately 0.15 inches) with fingertip control. The system was also able to compensate for changes to the robot's movements and surrounding environment.
“This work illustrates the transition from programming robots to robot education,” notes the doctoral degree. Sizhe Lester Li, student of MIT Web Features.
“Today, many robotic tasks require extensive engineering and coding. In the future, we expect that they will show the robot what to do and learn how to autonomously achieve their goals.”
Because this system relies solely on vision, it may not be suitable for lighter tasks that require contact sensing and tactile manipulation. That performance can also be reduced if there are insufficient visual cues.
Researchers suggest that the addition of touch and other sensors could allow robots to perform more complex tasks. It also has the potential to automate control of a wider range of robots, including those with embedded or unembedded robots.
Written for you by author Charles Blue, edited by Sadie Harley, and fact-checked and reviewed by Robert Egan. This article is the result of the work of a careful human being. We will rely on readers like you to keep independent scientific journalism alive. If this report is important, consider giving (especially every month). You'll get No ads Account as a thank you.
detail:
Sizhe Lester Li et al., controlling a variety of robots by inferring Jacobian fields in deep networks; Nature (2025). doi:10.1038/s41586-025-09170-0
©2025 Science X Network
Quote: The Deep Learning System teaches moving soft, bio-style robots using only one camera (2025, July 9) obtained from https://news/2025-07 on July 13, 2025.
This document is subject to copyright. Apart from fair transactions for private research or research purposes, there is no part that is reproduced without written permission. Content is provided with information only.
