PROWBench

Do Video Models Render What the Program Specifies?

Alaya Lab

TL;DR

PROWBench checks whether generated videos faithfully realize program-specified entity trajectories, camera motion, and interactions, using replayable world states and timestamped events as the reference.

Takeaways

  1. Proxy conditioning improves entity control over camera conditioning alone, with strong video priors also improving event alignment.
  2. Proxy videos already encode camera motion, allowing camera control comparable to several models given explicit camera poses.
  3. General-purpose video models are competitive with specialized renderers on short clips across control and event fidelity.
  4. Fine-tuning offers mixed gains, while reliable domain-specific events still benefit from dedicated training.
  5. Better proxies need to bind identity, appearance, orientation, and articulation as well as position and motion.
  6. Long-horizon entity reappearance and identity consistency across independently generated views remain unresolved.
  7. Interaction Success and Logic-Render Alignment reveal failures visual and spatial scores miss, but judge calibration remains a limitation.

Leaderboard

Compare model results across five evaluation settings. Enable JavaScript to explore the table, or read the project paper.

Structural inputs

Coarse 3D, CWM proxy, and Colored-OBB are rendered from the same recorded world state and camera trajectory. The world record retains entity states and events even outside the camera’s field of view.

Coarse 3D ↗

CWM proxy ↗

Colored-OBB ↗

0.0 / 5.0 s

Press Play to load this example.

Scenes and interactions

40 Verified-FF appearance references

First-frame appearance reference: People and cars at a sunlit city street crossing (Urban; third person view)
Crossing a street
First-frame appearance reference: A person beside a red car on a city street lined with buildings (Urban; third person view)
City driving
First-frame appearance reference: A person and cars sharing a broad tree-lined city street (Urban; third person view)
Navigating traffic
First-frame appearance reference: A person at the top of outdoor steps overlooking a residential street (Urban; third person view)
Descending steps
First-frame appearance reference: Two people with a small box on a wet residential street (Urban; third person view)
Passing an object
First-frame appearance reference: First-person view over a wooden cart on a cobbled old-town street (Urban; first person view)
Pushing a street cart
First-frame appearance reference: A worker with a wooden crate at an outdoor construction site (Urban; third person view)
Moving a crate
First-frame appearance reference: First-person hands behind a wooden box in a sunlit industrial interior (Indoor; first person view)
Pushing a wooden box
First-frame appearance reference: A seated person facing three people in a room rendered in an ink-wash style (Indoor; third person view)
Sitting down
First-frame appearance reference: People and a cart sharing a glass-fronted office corridor (Indoor; third person view)
Navigating a crowd
First-frame appearance reference: A person near warehouse stairs and a raised loading platform (Indoor; third person view)
Carrying upstairs
First-frame appearance reference: A person carrying a chair through an office corridor with other people nearby (Indoor; third person view)
Carrying a chair
First-frame appearance reference: A person with a shopping cart among store shelves and other shoppers (Indoor; third person view)
Pushing a shopping cart
First-frame appearance reference: A person approaching steps at a modern building entrance (Indoor; third person view)
Climbing stairs
First-frame appearance reference: People on a rocky mountain trail in an ink-wash landscape (Nature; third person view)
Walking on rocky terrain
First-frame appearance reference: People and a vehicle on an icy path through a snowy forest (Nature; third person view)
Sliding on ice
First-frame appearance reference: Cockpit view over a coastal town with an aircraft ahead (Sky; first person view)
Helicopter flight
First-frame appearance reference: Cars on a racing track bordered by pine trees (Racing; third person view)
Racing through a forest
First-frame appearance reference: Two cars beneath a track gantry on a reflective wet racing circuit (Racing; third person view)
Racing on a wet track
First-frame appearance reference: First-person view of people along a path through a stylized cave (Fantasy; first person view)
Exploring a cave
First-frame appearance reference: People on a path through a misty forest with oversized trees (Fantasy; third person view)
Exploring a fantasy forest
First-frame appearance reference: People among cacti and unusual rock formations in a stylized desert (Fantasy; third person view)
Exploring a desert
First-frame appearance reference: A dog on a snow-covered forest path (Nature; first person view)
Dog on a snowy trail
First-frame appearance reference: A cat crossing a sunlit living-room floor (Indoor; first person view)
Cat in a sunlit room
First-frame appearance reference: A rabbit on a wet paved park plaza (Urban; first person view)
Rabbit in an urban park
First-frame appearance reference: A person beneath flying birds in a grassy field (Nature; third person view)
Birds over grassland
First-frame appearance reference: A duck on a rain-soaked plaza beside trees and benches (Urban; first person view)
Duck on a rainy plaza
First-frame appearance reference: A diver among colorful fish, kelp, and coral underwater (Underwater; third person view)
Diving among fish
First-frame appearance reference: Cars on a forest race track beneath an illuminated gantry at dusk (Racing; third person view)
Racing at dusk
First-frame appearance reference: A stylized character with a wooden crate on a street beside workers and a van (Urban; third person view)
Pushing a crate
First-frame appearance reference: A seated person facing other people in a dark room lit by overhead lamps (Indoor; third person view)
Sitting in a dark room
First-frame appearance reference: A person pushing a shopping cart through a supermarket aisle (Indoor; third person view)
Pushing a cart indoors
First-frame appearance reference: A stylized character and cats on a quiet residential street at sunset (Urban; third person view)
Cats on a quiet street
First-frame appearance reference: A person and cats on a wet walkway beside trees and park benches (Urban; third person view)
Cats in a city park
First-frame appearance reference: A stylized character and rabbits leaving tracks on a snowy forest path (Nature; third person view)
Rabbits in the snow
First-frame appearance reference: A person and ducks on a wet park walkway bordered by geometric trees (Urban; third person view)
Ducks on a park walkway
First-frame appearance reference: A stylized person and ducks on a residential street with colorful houses (Urban; third person view)
Ducks in a neighborhood
First-frame appearance reference: First-person view of a duck in grass beside a fence and a country house at dusk (Nature; first person view)
Duck in the countryside
First-frame appearance reference: A person and ducks on a snow-covered path among trees and houses (Nature; third person view)
Ducks on a snowy path
First-frame appearance reference: A diver and fish above a sandy seabed with rocks and sparse aquatic plants (Underwater; third person view)
Diving over the seabed