The infant tests are as follows: Show me a glass of water on the desk. Hide behind a wooden plank. Move the board towards the glass. If the board continues to pass through the glass, are they surprised as if they weren't there? Many six months after birth, and by one year, almost every child has an intuitive concept of the persistence of objects learned through observation. So is the case with some artificial intelligence models now.
Researchers have developed AI systems to learn about the world via video, and have presented the concept of “surprise” when presented with information that contradicts the knowledge they have gathered.
The model created by Meta and called the Video Joint Embedded Prediction Architecture (V-JEPA) does not assume the physics of the world contained in video. Nevertheless, it allows us to begin to understand how the world works.
“Their arguments are a priori, very plausible, and the results are very interesting,” says Micah Heilbron, a cognitive scientist at the University of Amsterdam, who is studying how brains and artificial systems mean the world.
Higher Abstraction
As engineers who build self-driving cars know, ensuring that they understand AI systems can be difficult. It either categorizes most systems designed to “understand” videos (e.g. “Tenniser”), or identify the outline of an object, or works with, for example, what the car in front of you is called “pixel space.” This model essentially treats every pixel in the video as important.
However, these pixel space models have limitations. Imagine trying to understand the streets in the suburbs. If the scene has cars, traffic lights, or trees, the model may focus too much on unrelated details such as leaf movement. You may miss the color of the traffic light or the location of a nearby car. “When you go to pictures or videos, you don't want to work at [pixel] Randall Barestriello, a computer scientist at Brown University, said:.
Yann Lecun, a computer scientist at New York University and director of AI research at Meta, created Jepa, the predecessor of V-JEPA, which works with still images, in 2022.
Ecole Polytechnic University Parisa Clay
Released in 2024, the V-JEPA architecture is designed to avoid these issues. The details of the various artificial neural networks that make up V-JEPA are complex, but the basic concepts are simple.
A normal pixel space system undergoes a training process that masks several pixels into a video frame and trains a neural network to predict the values of these masked pixels. V-Jepa also masks some of the video frames. However, it does not predict what is behind the masked region at the level of individual pixels. Rather, they use a higher level of abstraction, or “potential” representation, to model content.
The latent representation captures only important details about the data. For example, given a line drawing of various cylinders, a neural network called an encoder can learn to convert each image into numbers that represent the basic aspects of each cylinder, such as its height, width, direction, and location. In doing so, information contained in hundreds or thousands of pixels is converted into a potential representation of a few numbers. Another neural network, called a decoder, learns to convert the essential details of a cylinder into an image of the cylinder.
V-Jepa focuses on creating and reproducing latent representations. At high levels, the architecture is split into three parts: Encoder 1, Encoder 2, and Predictor. First, the training algorithm takes a series of video frames, masks the same set of pixels for all frames, and feeds the frames into encoder 1. Encoder 1 converts masked frames into latent representations. The algorithm also feeds the entire unmasked frame into Encoder 2. Encoder 2 converts it to another set of latent representations.
Now the predictor gets caught up in the law. Predict the output of encoder 2 using the latent representation generated by encoder 1. Essentially, it takes a latent representation generated from a masked frame and predicts a latent representation generated from an unpopulated frame. By recreating related latent representations rather than missing pixels in the previous system, the model learns to see cars on the road and doesn't make a fuss about the leaves.
“This allows the model to discard unnecessary things, discard and focus information on more important aspects of the video,” says Quentin Gallid, research scientist at Meta. “It's extremely important to dispose of unnecessary information and is what V-Jepa aims to do so efficiently.”
Once this pre-training phase is complete, the next step is to adjust the V-JEPA to accomplish specific tasks such as image classification and identification of actions drawn on the video. This adaptation stage requires some human sign data. For example, information about the actions contained in the video must be tagged. Adaptation to the final task requires much less label data than if the entire system was trained end-to-end on a particular downstream task. Additionally, the same encoder and predictor networks can be adapted to a variety of tasks.
Intuitive imitation
In February, the V-JEPA team reported how the system did it by understanding intuitive physical properties in the real world, such as object persistence, shape and color constancy, and the effects of gravity and collision. In a test called intphys, you need to identify an AI model to identify whether the action taking place in the video is physically plausible or unbelievable. The VJEPA was almost 98% accurate. The well-known models predicting in pixel space were a little better than coincidence.
Autonomous robots need something like physical intuition to plan their movements and interact with the physical environment.
Wladimir Bulgar/Science Photo Library
The V-JEPA team also explicitly quantified the “surprise” presented by the model when its predictions did not match the observations. They adopted a V-JEPA model pre-processed with natural video, fed new videos, and mathematically calculated the difference between what V-Jepa was expecting in future frames of the video and what actually happened. The team discovered that prediction errors occur when future frames contain physically impossible events. For example, if the ball rolled behind an occluding object and temporarily disappeared from view, the model generated an error when it did not reappear from behind the object in a future frame. The response was similar to the intuitive response seen in infants. V-Jepa was surprised.
Heilbron is impressed by V-Jepa's abilities. “We know from the developmental literature that babies don't need much exposure to learn these types of intuitive physics,” he said. “It's convincing that they show that it's learnable in the first place, and you don't have to come in all of these innate advance.”
Carl Friston, a computational neuroscientist at the University of London College, believes V-Jepa is on the right track in mimicking “how to learn the brain, learn and model the world.” However, there are still some basic elements. “What are you missing? [the] For example, if information from past frames is not sufficient to accurately predict future frames, the prediction is uncertain and if V-JEPA does not quantify this uncertainty, the current proposal is a proper encoding of uncertainty.
In June, Meta's V-Jepa team released the 2.2 billion parameter model V-Jepa 2, pre-processed with 22 million videos. They also applied the model to robotics. They demonstrated how to further fine-tune the new predictor network using approximately 60 hours of robot data (including information about the robot's video and its actions), and used the fine-tuned model to plan the next action for the robot. “Models like this can be used to solve simple robotics manipulation tasks and pave the way for future work in this direction,” Galido said.
To push the V-Jepa 2, the team intphys2. We designed a more difficult benchmark for intuitive understanding of physics called V-Jepa 2. Garrido said the V-Jepa 2 can only process video for a few seconds as input, and it can be predicted in seconds in the future, Garrido said. No more has been forgotten. You can make another comparison with the toddler, but Galido had another creature in mind. “In a way, the memories of models are reminiscent of goldfish,” he said.
