One handheld video is now enough to rebuild a person in 4D
Figure: Jin et al., Zhejiang University / Robbyant / Ant Group / HKUST · Research
Yudong Jin, Tao Xie and colleagues at Zhejiang University's State Key Lab of CAD&CG, with Robbyant, Ant Group and HKUST, have built 4DAnyone, which takes a casually captured monocular video with unknown camera intrinsics and poses, generates the tens of multiview-consistent videos a 4D gaussian splatting reconstruction needs, and lifts them into a model you can orbit.
Why it matters: Free-viewpoint video of people has been a rig problem. DNA-Rendering, the benchmark this paper reports on, was captured with 48 synchronised cameras. That is the standard setup, and it is the reason volumetric human capture stayed inside studios that could afford one.
The alternative has been to generate the missing views with a video diffusion model, which produces footage that looks right and reconstructs badly. The gap between plausible and reconstruction-grade is where this work lives.
The bottleneck they named: The failure has a specific shape, and naming it is most of the contribution. A diffusion transformer can only hold so many target views in one forward pass. Past that, the views get split into groups, and two things break at once: conditioning each group on every previously generated view grows as O(N), which dilutes appearance guidance, and disjoint groups cannot see each other at all, which lets global structure drift apart.
Reference Context Packing compresses the growing set of reference views into a fixed-length mixed-resolution context, taking that first cost to O(1). Target Context Routing handles the second by rotating which views share a group as denoising proceeds — cycling the four-view groups at high noise, when global structure is being decided, then holding them fixed at low noise so neighbouring views can settle detail together.
By the numbers:
- 24.33 dB PSNR for generated-video consistency on DNA-Rendering, against 21.47 for a fine-tuned ReCamMaster given the same modules and data, 21.25 for MV-Performer and 13.56 for TrajectoryCrafter.
- 24.15 dB on the 4DGS reconstruction itself, against 20.55 for the strongest baseline — the consistency gains carry through the lift rather than washing out.
- 23.28 dB on DyMVHumans, which is out of distribution for every method tested.
- The ablation moves 21.09 → 22.63 dB, with roughly equal credit to each of the two mechanisms and a sliding group schedule beating random and strided.
- MVGameHuman, the dataset they built for this in an in-house game engine: 38,000 videos, 24 cameras, 318 actors at 2560×1440.
Yes, but: The training bill is not casual even if the capture is. The model was trained on 128 H20-3E GPUs against a purpose-built game-engine dataset plus light-stage and in-the-wild footage. What moved to the phone is inference, not the work behind it.
The evaluation is also narrow: 10 DNA-Rendering sequences and 3 from DyMVHumans, thirteen in total. And the numbers thin out at the edge of the distribution — generated-video reconstruction on DyMVHumans lands at 21.03 dB, several dB below the in-distribution figures, which is the honest measure of how far in-the-wild generalisation actually reaches today.
The big picture: The teaser figure is a person filmed on a phone in one room, surrounded by a ring of generated viewpoints that never existed. The back of her jacket is a hallucination consistent with the front, and the paper says so plainly.
Which is the interesting property and the uncomfortable one. A capture rig records what was there. A system like this decides what was probably there, at a fidelity good enough to reconstruct and re-render, from footage anyone can shoot without the subject noticing the camera was doing anything unusual.
