VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
What it is
VibeVideoBench is a benchmark for evaluating multimodal agents on their ability to generate video by writing code to render frames and synthesize audio, rather than using diffusion models. It consists of 239 self-contained tasks in English and Chinese that require agents to produce a finished MP4 file within a one-hour time limit in a sandboxed environment.
The problem it solves
Diffusion models struggle with precise, rule-consistent content (like formulas or charts), and there was no standardized way to evaluate whether agents could reliably write code to generate high-quality, synchronized video within strict time constraints.
What you could use it for
- Benchmarking the video generation capabilities of frontier LLM agents
- Evaluating agent planning and time management skills in long-horizon tasks
- Comparing the quality of code-driven video generation across different model providers
Why it matters
Provides a standardized, evidence-based metric for assessing the practical utility of autonomous agents in creative media production, specifically highlighting the gap between code execution capability and final delivery reliability.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.