The sight of robots walking down the street, surrounded by astonished onlookers, is becoming increasingly common. But these machines aren’t the do-it-all assistants you need to work in your kitchen or factory, and the big bottleneck is data. Like humans, robots learn best through experience. The challenge is that physically teaching these machines so many actions in different settings takes a lot of effort and time.
“One natural idea is to use simulation as a training ground. The physics engines that power robot simulators have made great strides in recent years, but one of the challenges that remains is creating simulation content that is rich and diverse enough to capture the complexities of the real world,” said Russ, Toyota Professor of Electrical Engineering and Computer Science (EECS), Aerospace, and Mechanical Engineering at MIT and Principal Investigator in the MIT Department of Computer Science and Mechanical Engineering. Teddrake said. Artificial Intelligence Laboratory (CSAIL).
It turns out that AI agents, semi-autonomous programs that “think” and complete well-defined tasks, can help generate the life-like virtual settings needed for robots. The new “SceneSmith” system, developed by researchers at MIT CSAIL and Toyota Research Institute, uses three agents to stitch together objects, walls, and the overall appearance of a 3D scene. Reproductions of indoor spaces such as restaurants, bedrooms, and hotels are more realistic and detailed than previous systems, helping the robots practice skills and try out different ways of doing tasks before turning them on. As a result, engineers save time on actual testing.
The agents know what everyday places look like because they each call a multimodal system called a Vision Language Model (VLM), specifically the state-of-the-art VLM GPT-5.2. It is trained using large amounts of text and images from the internet to handle more visual prompts. This advanced model gives each agent a type of spatial knowledge. First, a “designer” agent generates the elements of the scene, then a “critic” agent advises whether it looks realistic, and finally an “orchestrator” manages the interactions and decides when the design is complete. Once the three VLMs have finished their creative collaboration, the scene is ready to be loaded directly into the physics simulation software.
“We found that this system can construct 3D scenes in the same way that human designers do,” said Nicholas Pfaff, CSAIL researcher and MIT EECS doctoral student, lead author of the paper in which Tedrake published the research. “We created over 1,300 scenes using a leading VLM with internet-scale priors, and the arrangements were very creative and varied. We didn’t tell the system to do it with a prompt; we just improvised.”
Please talk to your agent
Thanks to the VLM agent, you can ask SceneSmith to do things like “generate a garage with a car, a workbench, a tire stacked in the corner, and a ladder on the wall,” giving you a virtual playground full of objects for your robot to play with. These rooms are decorated with up to six times more items per scene than traditional methods and are perfect for the robot to learn skills like putting a cup in the sink, fruit on a plate, and moving a can of soda from a shelf to a table.
A rich array of virtual environments allows you to assess whether your robot is ready for deployment without having to go through trial and error in the physical world. The researchers tested different action plans (also known as “policies”) in SceneSmith’s digital world, generating 100 unique spaces in the process. When the VLM agent evaluated each attempt, it discovered that the robot’s plan was flawed and the machine often failed the chore. Humans agreed with the model’s decisions more than 99 percent of the time, potentially helping roboticists weed out flawed approaches in simulations before robots move in the real world.
But how real are these virtual worlds, in fact? It can be difficult to prove completely, so researchers approached the question from several angles. The most important test was dropping a pre-trained robot policy (an AI controller that had never seen a SceneSmith scene and was trained primarily on real-world data) into the generated environment. In one test, a user told the system to “take an apple out of the bowl and place it on the cutting board,” and a simulated robot did just that. If the scene didn’t closely resemble the actual setting the policy was learned from, it wouldn’t have worked at all.
The team also remotely controlled the robot through virtual space to open cabinets, put away bottles, and guide it between rooms. Their experiments revealed that the environment persists under sustained physical interaction and extends beyond visual inspection.
behind the scenes
Each agent SceneSmith uses has a well-defined role in the generation process, gradually fleshing out the scene. They basically create a floor plan and make it happen.
Let’s say you want to create a scene that resembles the first floor of a house. A “designer” VLM starts with a general layout, which is reviewed by a “critic” and then approved by an “orchestrator.” The agent repeats this approach at each step. Add furniture, place objects on the walls, ceiling, and finally place objects that the robot can manipulate. For example, VLM can add cabinets that robots can open and close, articulated items that weren’t common in the previous baseline.
At each stage, the second VLM checks that the scene is practical and advises, for example, to remove the bathtub from the living room. Third, VLM ensures that a high-quality scene is produced, even if you go back through the design process several times if the visuals aren’t up to par. Once the three VLMs have finished their creative collaboration, the mechanics of the physical world will be added through simulation software.
SceneSmith’s solid understanding of how a room should look, where objects should be placed, and real-world physics gives it a distinct advantage over traditional methods. Compared to scene generation baselines like “HSM” and “Holodeck,” SceneSmith created environments with more objects, such as a private office, a pottery store, and even a Minecraft-themed game room.
SceneSmith was also popular among over 200 users. They found that the system’s visuals were more realistic over 90% of the time. They also observed that, generally speaking, they followed the prompts more closely than other approaches. In other words, it was perfect for generating a virtual playground that users would actually want to see.
A system with many talents
Realism, variety, and richness are all suitable for SceneSmith, even when generating individual 3D objects. When you tell it to create a rotating serving cart, it creates a 2D image and turns it into a detailed model with physical properties such as mass, friction, and inertia.
However, such a detailed process comes with a speed tradeoff. It can take several hours to create a single scene, as agents create and closely inspect each object. Increasing computing power can significantly increase system efficiency. CSAIL engineers hope to expand to deformable objects (such as sponges) once an extensive 3D library becomes available.
“SceneSmith represents a significant advance in this regard by providing an agent framework for generating simulation-ready indoor environments from simple text prompts,” said Jeremy Binagia, an applied scientist at Amazon Robotics who was not involved in the research. “This advances the state of the art in several ways, including pushing the limits of object density in simulated environments, ensuring all objects are physically accurate (rather than just visually realistic), and creating assets that are not constrained by fixed libraries because they can be generated via text-to-3D conversion.”
Pfaff and Tedrake co-authored the paper with Thomas Cohn SM ’24, an MIT doctoral student and CSAIL researcher. Toyota Research Institute roboticists Sergey Zakharov and Rick Corey (SM ’08, PhD ’10); Their research was supported in part by Amazon, the U.S. Office of Naval Research, Toyota Research Institute, and the U.S. National Science Foundation.
The research team presented their findings as a spotlight at the International Conference on Machine Learning last week.
