The AI industry has cracked video generation. Models like Sora, VEO, and Kling allow you to produce footage on demand at very low cost. But Maor Bril, an engineer who worked at Character.ai for more than two years, says there are problems in the industry that no one wants to talk about. Most of the output is sloppy, full of broken physics, drifting characters, and disjointed storytelling. In a recent conversation on the AI Engineers podcast, he claimed that the hard part is that he doesn’t make videos anymore. It’s about knowing whether the video you made is really good or not.
“Put eval inside the generation loop,” Bril said. “Get eval as close to the generating loop as possible.”
Based on this principle of catching errors early, before they turn into expensive rework, his team built a video quality assessment tool that runs in about 3 seconds on a 15-second clip. To get there, we had to abandon our first attempt, redesign the way we trained based on data, and abandon our instinct to score videos on a scale of 1 to 10.
Why existing metrics are blind to storytelling
The AI industry already has tools to measure the quality of output. CLIP score checks whether a single frame matches a text prompt. LPIPS measures how far one frame deviates from the next. These simultaneously cover rapid coordination and temporal consistency. But Brill argues that video is “just another form of storytelling,” and combining these metrics doesn’t really answer the important question: Does the physics hold across multiple shots? Do the characters look the same from scene to scene? Is the pacing natural?
These are axes that exist only over time and throughout the shot. Ready-made indicators are blind to them. The team needed something that could be viewed as a complete story, rather than frame by frame, just like a human would.
The lure of an LLM as a judge — and why it wasn’t enough
The first systematic solution combined frame-level metrics with a large-scale language model to serve as the decision-maker. Human annotators provided calibration and scored the videos, and used that feedback to adjust LLM’s prompts so that its decisions more closely matched human preferences. This was a reproducible benchmark harness and was better than nothing.
It was also too slow, too costly, and shockingly inconsistent.
“LLM as a judge is slow and only as good as your prompts. If multiple people prompt in different ways, you will get completely different responses,” Brill pointed out. The same model can give very different ratings for the same video depending on how the question is worded. Even worse, the core question is: “Is it a good thing?” — inherently subjective and poorly defined. The team realized they needed something that was faster, cheaper, more consistent, and could be deployed closer to where the video was actually being produced.
Distilling the Commission into a 3-Second Judge
The answer was distillation. The team employed an expert metrics committee and calibrated LLM judges and used them to train a single small vision language model. Specifically, the team chose a tweaked version of Qwen because of its positive post-training experience. Other VLMs were tested, but Qwen had sufficient speed and accuracy.
The small model evaluates a 15 second video in about 3 seconds. The larger Frontier model had slightly better accuracy, but the speed advantage was decisive. In order to keep the evaluation within the production loop, it needs to be near real-time.
| stage | tool | Speed/characteristics |
|---|---|---|
| Committee evaluation | LLM and professional indicators as a judge | It’s too late. Sensitive to prompts. expensive |
| Extracted evaluators | Fine-tuned Qwen VLM | Approximately 3 seconds per 15 second video |
| human reconditioning | Regular annotation sessions | 10-15 minutes per session per employee |
Compare, don’t score
The most important design decision was a structural one: training the model to compare two videos rather than assigning an absolute score.
“Don’t score absolutely, but compare,” Brill said. “If you show them two videos and ask them which one tells the better story, the majority will agree.”
Humans are bad at agreeing whether something deserves a 6 or an 8, but we’re surprisingly consistent when asked to choose the better of two options. Training the model on pairs of video A and video B instead of labeled scores reduces subjectivity and improves generalization. This insight shaped everything downstream, including how we built our training datasets, how we structured our loss functions, and how our models reported their decisions in production.
Manufacturing fraud and the trap of AI detection
There are plenty of great videos on the internet. Bad videos, very few helpful videos pointing out specific flaws. The team had to create it intentionally. They corrupted high-quality footage by adding artifacts and destroying physics, and generated random slop from scratch.
This process involved hidden risks. If the training data consistently combines real-world footage with AI-generated footage, the model may learn to detect “AI-likeness” rather than actual quality. Shallow shortcuts become useless for evaluating truly good AI output. This fix required careful parity, including consistent encoding, consistent annotation methods, and systematic pairing to prevent model cheating.
V1 Disaster: Score atmosphere, not video
In Brill’s words, the evaluators’ first version was “very wrong, absolutely wrong.”
I gave it a camerawork score of 9.2 for a static 4-second frame where the camera wasn’t moving at all. It celebrated physics with clips of ghosts floating and people flying. The model learned to recognize things like consistency, visual gloss, and artificial sheen, but it wasn’t the defined axes of storytelling, physics, and pacing that the team had intended. It enlivened the atmosphere.
The root cause was in the training data. Without careful curation, the model stuck to superficial patterns that roughly correlated with quality, but missed the essence entirely. This fix required rebuilding the dataset from scratch by systematically combining real human footage and AI-generated footage while controlling encoding parity and annotation consistency.
From rigid pipelines to agent workflows
Early rating systems were built as fixed pipelines where videos were input and scores were output. This worked for narrow use cases, but not when users brought their own characters, images, and voices. Every story is different, and a static pipeline could not adapt.
The team moved to agent workflows. Instead of one monolithic evaluation step, the system consists of agents with tools (including miniature VLMs) that can verify output, validate their own work, and adapt as the user’s narrative evolves. When the agent detects a character shift between two adjacent starting frames, it can trigger the regeneration of a single 5-6 second clip before it is pasted into the final full-length video. The cost of fixing a bad clip is a fraction of the cost of fixing a finished film.
What’s still broken: Audio and lip sync
Audio evaluation is performed using a combination of techniques. The team uses Atmos to measure audio quality and clarity. VLM associates keyframe timestamps with sound spikes. For example, verify that the peak of the sound of a door closing matches the exact frame that the door touches.
However, the lip sync issue remains unresolved. In conversation animation, mouth movements do not have a natural correlation with speech, and current methods make it difficult to assess whether lip movements match dialogue. Bril flagged this as an active area of work without a clear solution yet.
Keeping judges aligned with humans over the long term
Taste changes. What looks impressive today may look outdated in six months. To prevent drift, the team runs regular annotation sessions (10-15 minutes per employee) and scores random videos on individual axes. The results recalibrate the LLM-as-judge component and feed the annotated data into the training set of the next version of VLM. Bril acknowledges that this process is not instantaneous and taste is inherently subjective, but it provides a mechanism to evolve the model as standards change.
The methodology Brill describes comes at a time when AI-generated content is facing unprecedented scrutiny. Nvidia’s AI rendering technology for games, DLSS 5, came under fire earlier this year for creating “AI slop” after early demos showed unusually smooth facial details on characters. In response to a New York state law requiring transparency around “synthetic performers,” Amazon just started requiring third-party sellers to disclose when their product images feature AI-generated people. And across Wall Street, Nomura analysts are warning that it’s becoming increasingly difficult to measure returns on AI investments: the core question: “Is this good?” This applies not only to pixels but also to business cases. For engineers building the next generation of video tools, Bril’s framework (comparison rather than scoring, defining clear narrative axes, evaluating in the loop) provides a template for turning subjective quality into something important that machines can measure quickly enough.
