PROWBench

Do Video Models Render What the Program Specifies?

TL;DR

PROWBench checks whether generated videos faithfully realize program-specified entity trajectories, camera motion, and interactions, using replayable world states and timestamped events as the reference.

Benchmark contents

Evaluation cases: 170
Selected views: 200
Proxy videos: 600

Tracks and settings

Main track

Verified-FF

40 pairs

5 s · 1 view

A shared RGB first frame, checked against the recorded state.

Main track

Unverified-FF

90 pairs

5 s · 1 view

Appearance and identity binding from text and control signals.

Challenge track

Long horizon

10 cases

30 s · 1 view

Entity reappearance and state persistence across generation windows.

Challenge track

Multi-view

10 cases

5 s · 4 views

Four synchronized cameras, with each view generated independently.

Plus 20 Gunfight cases from 10 recordings for domain-specific evaluation.7 domains · 30 environments · 47 semantic actions · 9 interaction categories.

Metrics

Interaction Success Rate: VLM-based evaluation of whether each recorded event shows both its prescribed action and expected end state.
Logic-Render Alignment: VLM-based evaluation of whether each prescribed action appears within its engine-annotated time segment.

gISR / gLRA multiply each sample’s behavioral score by its BBox IoU, then average across samples.The suite also measures entity and camera control, visual and temporal quality, multi-view consistency, and long-horizon memory.

What do Verified-FF and Unverified-FF mean?

The main-track settings use disjoint scenes. Verified-FF supplies the same MiniMax-H3-generated first frame to every method that accepts one, after checking it against the recorded person count, positions, facing, and layout.

Unverified-FF supplies no prescribed appearance reference. Methods whose interface requires an RGB image use unchecked frame 0 of the MiniMax-H3 output for that scene and camera. This can transfer H3’s appearance assignments; the paper also notes potential curation bias from using H3 outputs to select this set.

Structural inputs

Coarse 3D, CWM proxy, and Colored-OBB are rendered from the same recorded world state and camera trajectory. The world record retains entity states and events even outside the camera’s field of view.