Generating Compositional Worlds
from Grounded Videos
△ Alaya Lab ♣ The University of Tokyo
Correspondence: Zhixiang Wang (Project Lead), Kaipeng Zhang
Given RGB images with instance masks and 3D boxes, WorldSculpt produces a compositional mesh representation for very complex scenes consisting of hundreds of individual objects.
We propose WorldSculpt, a framework that reconstructs cluttered scenes of hundreds of objects into complete per-object meshes.
We modify Pixal3D, a single-object generative prior, to consume occluded multiple views, and leverage it for compositional scene generation.
We introduce a benchmark of densely-cluttered scenes with hundreds of objects, and show that the same pipeline turns generated 3DGS worlds (e.g., Marble) into compositional meshes.
Every object is an individual mesh. Hover to highlight an object, click to isolate it, and drag the slider to pull the scene apart. This is what compositional means.
The GLBs shown here are post-processed and compressed for web viewing. For the original files, refer to the GitHub repository.
We run WorldSculpt on 3D Gaussian Splatting scenes generated by World Labs' Marble: from rendered views of each generated 3D Gaussian Splattings scene, WorldSculpt reconstructs instance mesh for every object — turning a radiance field into asset-ready geometry. The source 3DGS scene (left) and the reconstructed meshes (right) share one synchronized camera — drag either side, or pull the explode slider to take the parts out of the world.
The 3DGS layer is pruned to 800k gaussians for web viewing. The meshes are post-processed and compressed for web viewing. For the original files, refer to the GitHub repository.
WorldSculpt inherits the object-level prior of Pixal3D and equips it with a spatially-grounded, multi-view conditioning pathway — per-view DINOv3 features are lifted into the object's canonical voxel grid and fused by a permutation-invariant IBR-style aggregator, injected through zero-initialized layers and LoRA.
Each object's 3D box defines a canonical cube; the anchor view fixes its orientation. Posed scene images are cropped, masked, and mapped into the virtual canonical space the generative prior expects.
DINOv3 features from every view are lifted into the canonical voxel grid by off-center projection and fused with a learned, permutation-invariant aggregator — so occluded regions are plausibly synthesized while observed regions stay faithful.
Every generated mesh is complete and self-consistent, so composing the scene reduces to rigid transforms — no cross-object fusion, no per-object post-hoc optimization. Trained only on single canonical objects, it scales to hundreds per scene.
Six photorealistic environments rendered in Unreal Engine 5.8 at 2560×1440, densely populated with up to several hundred assets in natural arrangements — with complete per-object meshes, camera poses, instance masks, 3D boxes, and metric depth.
Due to copyright restrictions we cannot release the GT meshes. To evaluate on the benchmark, please contact Zhixiang Wang (zhixiang.wang@shanda.com).
@article{niu2026worldsculpt,
title = {WorldSculpt: Generating Compositional Worlds from Grounded Videos},
author = {Niu, Muyao and He, Jixuan and Yu, Ruihan and Fu, Lian and Huang, Zheng-Hui and Zhan, Yifan and Lan, Fengbo and Ge, Yongtao and Zheng, Yinqiang and Zhang, Kaipeng and Wang, Zhixiang},
journal = {arXiv preprint},
year = {2026}
}