BEAST3D is a self‑supervised framework that learns 3D visual representations from unlabeled, calibrated multi‑view video of animals. It uses a vision transformer to predict 3D Gaussian splats, reconstructing held‑out views via differentiable rendering while simultaneously segmenting the animal. The method works with as few as four views, produces viewpoint‑invariant features, and transfers to tasks such as novel view synthesis, multi‑view pose estimation, and neural encoding across four species.
By Yanchen Wang, Lenny Aharon, Wangshu Zhu, Kyle Daruwalla, Linghua Zhang, Jiaru Zou, Selmaan Chettih, Helen Hou, Liam Paninski, Matthew R Whiteway
arXiv:2503.10212v3 Announce Type: replace
Abstract: Analyzing animal behavior is crucial in neuroscience, yet deciphering its intricate dynamics remains challenging. Traditional machine vision approa...
By Teng Xu, Taotao Zhou, Youjia Wang, Peng Yang, Simin Tang, Kuixiang Shao, Zifeng Tang, Yifei Liu, Xinyuan Chen, Yihan Liu, Zhengning Xu, Boyang Xia, Hongshuang Wang, Xiaohui Wang, Huoqing Luo, Jingya Wang, Ji Hu, Jingyi Yu
World models enable agents to perform forward rollout and planning without real-world interaction. However, their application in open-world embodied intelligence remains limited by the high cost of action annotations and the heterogeneity of action spaces across platforms.
The paper introduces a biologically inspired framework that learns object‑centric visual representations from raw videos without human annotations or camera calibration. By using motion boundaries detected via optical flow and clustering to create pseudo‑instance masks, the method supervises a single‑image encoder with pixel‑level pairwise metric learning. Training on 195 million pseudo‑labeled frames and expanding to 421 million frames through Motion‑Verified Self‑Training, the approach yields Swin‑based encoders that outperform or match supervised and self‑supervised baselines on tasks such as monocular depth estimation, 3D object detection, 3D occupancy prediction, and end‑to‑end planning.
By Boshi Li, Xiaohui Wang, Xiaoyang Wu, Zhichao Li, Ya Yang, Naiyan Wang
arXiv:2609.38347v1 Announce Type: new
Abstract: Quantifying collective fish behavior requires accurate trajectories, yet multi-view 3D tracking remains challenging due to frequent occlusions, visuall...
By Patt Phurtivilai, Zhiyang Dou, Yifan Wu, Kinfung Chu, Yuan Liu, Lei Yang, Wenping Wang, Taku Komura
ORMA is a training‑free framework that reconstructs articulated 4D representations of animals from monocular videos by decoupling pose and shape. It uses predicted pose as a reference for optimization and generative 3D priors to refine shape, aligning the result with the SMAL+ parametric model. The method combines per‑frame pose estimates with globally consistent camera poses, and further refines the reconstruction using self‑supervised DINO correspondences and temporal consistency, achieving improved accuracy on the new PAW4D benchmark and diverse real‑world videos.
By Xuyi Hu, Francesco Palandra, Shangzhe Wu, Daniel Cremers, Riccardo Marin, Silvia Zuffi
Neural Voxel Dynamics introduces a self‑supervised framework that learns 3D latent dynamics from monocular video by unprojecting V‑JEPAs features into a voxel grid using depth priors. It then applies Volumetric Feature Advection, an action‑conditioned transition model, to predict the evolution of these latent features directly in 3D, without relying on privileged physics‑engine states. The method is evaluated on synthetic and real datasets, demonstrating predictive fidelity, 3D geometric consistency, and physical plausibility via Physics‑IQ metrics.
By Zican Wang, Niloy Mitra
BAL (Bidirectional Autoregression in Latent Space) is a self‑supervised framework that learns human movement representations by predicting future and past latent states, mirroring how humans anticipate actions. It decomposes predictions into global sequence‑level semantics and local context‑independent movement, ensuring non‑trivial training signals and preventing collapse. Experiments on large datasets show BAL outperforms baselines in motion captioning, forecasting, and interpolation tasks.
By Genki Kinoshita, Ko Nishino
arXiv:2606. 07687v1 Announce Type: cross Abstract: Video world models are increasingly used to provide predictive visual representations, yet it remains unclear which pretraining signals induce action-relevant structure in their latent spaces.
By Jewon Yeom, Hanseul Kim, Jeongjae Park, Sungmok Jung, Jaejin Lee, Taesup Kim
arXiv:2609.19142v1 Announce Type: new
Abstract: World models endow perceptual systems with the ability to predict how scenes evolve under interaction. They are most beneficial when trained on diverse...
By Bardienus P. Duisterhof, Kaifeng Zhang, Adam Hung, Bowen Wen, Stan Birchfield, Yunzhu Li, Deva Ramanan, Jeffrey Ichnowski
OnlyAct proposes extracting action representations from a pretrained foundation model (V-JEPA 2) instead of learning actions from scratch. It uses a label‑free contrastive adaptation framework to isolate motion while discarding scene information, avoiding reconstruction bottlenecks. The method is evaluated on diverse benchmarks, showing more expressive action representations and better transfer of actions to new scenes.
By Runjia Li, Minghao Chen, Junyu Xie, Philip Torr, Andrea Vedaldi, Tomas Jakab
DeepSTEP (STEP) is a Transformer-based framework that jointly performs tracking and pose estimation for animals and humans, eliminating the need for per-frame detections or post hoc matching. It uses discriminative model weight prediction to propagate identity across frames and introduces Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) to directly predict anatomically meaningful keypoints from an initial bounding box. Evaluated on diverse species datasets, STEP achieves strong performance and greater efficiency compared to existing pipelines.
By Shashikant Verma, Harish Katti, Soumyaratna Debnath, Yamuna Swamy, Shanmuganathan Raman