Grafy News
Try Grafy free

New Multi-Agent Framework Uses Self-Improvement for Video Evaluation

Researchers have introduced EVA-RSI, a framework that uses recursive self-improvement to fix motion defects in AI video evaluators without human labels.

Illustration for the news story: New Multi-Agent Framework Uses Self-Improvement for Video EvaluationSource: Grafy Research Lab

Researchers have developed a multi-agent framework designed to improve how AI models evaluate the temporal quality of video. By using a process called recursive self-improvement (RSI), the system aims to solve a persistent problem in video generation: evaluators often fail to detect motion defects, which in turn prevents video generators from learning how to produce smooth, physically accurate movement.

What is EVA-RSI and how does it work?

According to the research paper by Yu Yuan et al., EVA-RSI is a multi-agent framework that addresses two primary hurdles in video evaluation. First, unlike mathematics or coding, video evaluation lacks a "verifier"—a definitive way to check if an answer is objectively correct. Second, video quality is multidimensional, requiring a balance between physics, smoothness, and dynamics; improving one often degrades another.

To overcome the lack of a verifier, the researchers constructed a "proxy exam." They apply controlled, known changes to clean video clips—such as reversing the footage, dropping frames, or making objects disappear—to create synthetic supervision. This allows the system to know exactly what the "correct" answer should be without needing constant human labeling. The framework uses a team of agents: proposer agents suggest improvements, a leader agent coordinates these proposals, and an acceptance rule based on Nash bargaining ensures that new versions of the evaluator are only adopted if they improve key dimensions without falling below a minimum quality floor.

Why do current video evaluators struggle with motion?

Existing video evaluators often suffer from "blind spots" regarding temporal consistency. The authors note that even advanced vision-language models can fail significantly; for instance, they cited a case where a model assigned the same score to a normal video clip and an edited version where a person suddenly freezes.

Because video generators are often trained using these evaluators as reward signals in reinforcement learning (RL), any error in judgment is passed on to the generator. The researchers pointed out that using certain existing reward models can actually cause a generator to reduce motion over time, as the evaluator fails to properly incentivize complex movement. This creates a cycle where the generator produces increasingly static or physically unrealistic videos because it is being "graded" by a flawed judge.

How does the self-improvement process evolve?

The EVA-RSI process operates in iterative rounds. In each round, the system starts from the current "champion" model. The agents study the current evaluator's errors and propose new training recipes. A frozen checker then tests these new candidates against the proxy exam.

If a candidate passes the Nash bargaining acceptance rule, it becomes the new champion for the next round. The researchers also incorporated human involvement in a structured way: humans design the initial exam, review candidate changes, and revise the curriculum between major stages to ensure the agents do not drift too far from human perceptual standards. This hybrid approach combines autonomous agent search with human-guided curriculum design.

What are the results of using EVA-RSI?

Starting from the Qwen3-VL-8B model, the EVA-RSI framework demonstrated significant improvements across several metrics. The researchers found that the largest gains were seen in motion smoothness. Importantly, these improvements were not limited to the specific synthetic errors used in the exam; the evaluator's ability to detect defects transferred to types of motion errors it had never encountered during training.

When tested on public benchmarks, the EVA-RSI evaluator achieved the highest scores among 21 different models tested on dynamics subsets. Furthermore, when the improved evaluator was used as a reward signal for reinforcement learning, it successfully increased the amount of motion in generated videos and earned higher preference from human raters compared to previous methods.

How can this research be applied in practice?

For developers working on generative video models, this research suggests that the bottleneck for high-fidelity video is not just the generator, but the quality of the reward signal used to train it. Moving toward multi-agent, self-improving evaluation loops could provide a path to bypass the scarcity of human-labeled motion data.

While Grafy does not currently host a dedicated video training environment, researchers and developers can use Reader to analyze the full technical details of this paper, including the complex mathematical implementation of the Nash bargaining rule and the specific RL reward configurations used in the study.

Frequently asked questions

What is the main goal of EVA-RSI?

The main goal of EVA-RSI is to improve the ability of AI models to evaluate the temporal quality of videos. By using recursive self-improvement, the framework seeks to detect motion defects like freezing or unnatural physics, which current video evaluators often miss, thereby providing better feedback for video generation models.

How does the system solve the lack of a verifier in video?

The system solves the lack of a verifier by using a proxy exam with constructed targets. Instead of relying on human labels for every clip, the researchers apply known edits to videos—such as reversing them or dropping frames. Since the edits are controlled, the system knows the correct answer, providing the necessary supervision for the agents.

Does improving one dimension of video quality hurt others?

Yes, the researchers noted that video evaluation is multidimensional, meaning a change that improves smoothness might hurt physics or dynamics. To prevent this, EVA-RSI uses a Nash bargaining acceptance rule. This rule only accepts a new evaluator version if it improves the main dimensions or maintains them while ensuring no single dimension falls below a set minimum threshold.


Source: Grafy Research Lab — EVA-RSI: Recursive Self-Improvement of Video Evaluators for Judging Temporal Quality