Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos.
TAPVid-MV is a new benchmark for tracking any point in 3D across multiple synchronized camera views. It comprises 284 sequences, 1,142 calibrated camera streams, and 109,769 point tracks, covering indoor and outdoor domains and derived from various modalities such as depth, LiDAR, SLAM, and simulation. The dataset is visually verified, and evaluation shows that current multi‑view trackers do not consistently outperform monocular trackers, highlighting geometry recovery as a key bottleneck.
By Skanda Koppula, Frano Rajic, Abdullah Faiz Ur Rahman, Yi Yang, Ignacio Rocco, Jeet Thakwani, Rishabh Kabra, Andrew Zisserman, Joao Carreira, Siyu Tang, Carl Doersch, Gabriel Brostow
The paper introduces PLANET, a multi‑object tracker that transcends traditional image‑plane limitations by incorporating 3D scene geometry into its query formation. By lifting 2D tracking datasets into 3D and embedding reconstructed geometry into features and positional encodings, PLANET encourages queries to encode object positions. An auxiliary 3D location prediction task and a dual‑resolution temporal memory further enhance performance, enabling state‑of‑the‑art results on three diverse benchmarks.
By Orcun Cetintas, Guillem Bras\'o, Tim Meinhardt, Laura Leal-Taix\'e
TrackEverything is a 3D point tracker that overcomes the trade‑off between sparse long‑horizon tracking and dense short‑clip tracking by representing videos as persistent 3D scene tracks in world coordinates. It introduces voxel‑based de‑duplication at sliding‑window boundaries, a two‑stage refinement process (endpoint refiner and lightweight trajectory refiner), and a 3D WAFT module that replaces memory‑heavy 4D correlation volumes with efficient feature sampling. The method can track all visible points in videos longer than 1000 frames using only 40 GB of GPU memory, outperforming existing dense trackers on short clips and matching sparse trackers on long sequences.
By Ayush Jain, Sreeharsha Paruchuri, Ishita Gupta, Fan Zhang, Tanner Schmidt, Jakob Engel, Katerina Fragkiadaki, Adam W. Harley
Monocular videos record 3D scenes as sequences of 2D image-plane projections, obscuring depth and spatial relationships. Multi-object trackers localize and associate objects primarily using appearance...
arXiv:2609.18363v1 Announce Type: new
Abstract: Online multi camera 3D tracking must maintain scene global identities across synchronized views, yet query-based trackers carry these identities only i...
By Pragyan Shrestha, Haruto Nakayama, Atom Scott
arXiv:2603.12064v3 Announce Type: replace
Abstract: We address the challenging problem of dense dynamic scene reconstruction and camera pose estimation from multiple freely moving cameras -- a settin...
By Shuo Sun, Unal Artan, Malcolm Mielle, Achim J. Lilienthaland, Martin Magnusson
Real-world spatial intelligence requires agents to understand scenes from continuous video streams, where objects move, persist, disappear, and reappear over time. While recent spatial foundation models have enabled generalizable feed-forward 3D reconstruction, most streaming methods remain geometry-centric and lack temporally consistent object-level understanding.
arXiv:2609.24487v1 Announce Type: new
Abstract: In this work, we present a method for shape reconstruction and tracking from video via agentic analysis-by-synthesis. Unlike prior methods which first...
By Kirill Mazur, Nikita Karaev, Matthew Chang, Jitendra Malik, Nur Muhammad "Mahi'' Shafiullah
arXiv:2609.38347v1 Announce Type: new
Abstract: Quantifying collective fish behavior requires accurate trajectories, yet multi-view 3D tracking remains challenging due to frequent occlusions, visuall...
By Patt Phurtivilai, Zhiyang Dou, Yifan Wu, Kinfung Chu, Yuan Liu, Lei Yang, Wenping Wang, Taku Komura
arXiv:2610.01286v1 Announce Type: new
Abstract: Recent depth foundation models like Depth Anything 3 (DA3) achieve remarkable multi-view depth estimation but assume static 3D scenes, limiting their a...
By Xinhao Xiang, Weiyang Li, Zhijie Zheng, Abhijeet Rastogi, Jiawei Zhang
GRACE is a camera‑efficient multi‑view pedestrian tracker that reduces the number of required cameras while maintaining high tracking accuracy. It combines volumetric‑guided fusion of homography‑based BEV features with 3D‑lifted features, uses ray conditioning to incorporate each camera’s viewing direction, and employs BEV Track Recovery to continue existing tracks with low‑confidence detections. On the WildTrack dataset, GRACE raises MOTA from 83.54 to 91.07 compared to the baseline TrackTacular.
By Taigo Sakai, Kazuhiro Hotta, Hiroki Kouno, Naoki Kato