The video models generated by early AI were full of obvious flaws. Models like the original version of Sora in Openai were surprisingly good, but I was barely able to go in 15 seconds before displaying the glitches that ruined the magic trick.
Newer models such as Runway's Gen-4 and Google's VEO 3 can create videos that are almost indistinguishable from reality. Part of realism comes from the ability of models to predict physical world movements, such as the way light reflects objects and the behavior of water.
The AI model is a “black box” and its internal mechanisms are like a mystery, so there is no way to explain exactly how it replicates real life.
For tasks like complex engineering or traditional weather forecasting, scientists use systems known as physical models to make decisions. These models require a huge amount of calculations to make very narrow predictions.
Neural networks can be used as a type of shortcut to approximate the work of physics models. Recently, companies have made great strides by using neural networks to predict parts of the physical world, such as weather and machinery operations.
However, the AI models used in video creation are not particularly trained to model physics. They do it as a by-product of ingesting huge amounts of videos.
There is evidence that Video Models doesn't merely mimic countless videos that are trained, Germanidis says.
Runway tests the model by asking you to generate a video of something simple, like a ball hanging from the ceiling, and comparing the trajectory of that generated object to a real-world version that acts as ground truth.
The model still makes mistakes and cannot replace traditional physics modeling. However, Germanidis said they were not approaching the plateau of ability. “We don't train models on most of the world's observations, such as the results of scientific experiments and sensor data available to us,” he said. “The moment we do that, we're going to build a more powerful simulator of the world.”
