arXiv Computer Vision

Sparse-View Interpretable 3D Animal Behavior Representations for Neural Encoding and Decoding

arXiv Computer Vision
Nov 6

BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting

BEAST3D is a self‑supervised framework that learns 3D visual representations from unlabeled, calibrated multi‑view video of animals. It uses a vision transformer to predict 3D Gaussian splats, reconstructing held‑out views via differentiable rendering while simultaneously segmenting the animal. The method works with as few as four views, produces viewpoint‑invariant features, and transfers to tasks such as novel view synthesis, multi‑view pose estimation, and neural encoding across four species.

By Yanchen Wang, Lenny Aharon, Wangshu Zhu, Kyle Daruwalla, Linghua Zhang, Jiaru Zou, Selmaan Chettih, Helen Hou, Liam Paninski, Matthew R Whiteway
arXiv Computer Vision
Oct 26

MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis

arXiv:2503.10212v3 Announce Type: replace Abstract: Analyzing animal behavior is crucial in neuroscience, yet deciphering its intricate dynamics remains challenging. Traditional machine vision approa...

By Teng Xu, Taotao Zhou, Youjia Wang, Peng Yang, Simin Tang, Kuixiang Shao, Zifeng Tang, Yifei Liu, Xinyuan Chen, Yihan Liu, Zhengning Xu, Boyang Xia, Hongshuang Wang, Xiaohui Wang, Huoqing Luo, Jingya Wang, Ji Hu, Jingyi Yu
arXiv Computer Vision
Sep 7

Object Concepts Emerge from Motion

The paper introduces a biologically inspired framework that learns object‑centric visual representations from raw videos without human annotations or camera calibration. By using motion boundaries detected via optical flow and clustering to create pseudo‑instance masks, the method supervises a single‑image encoder with pixel‑level pairwise metric learning. Training on 195 million pseudo‑labeled frames and expanding to 421 million frames through Motion‑Verified Self‑Training, the approach yields Swin‑based encoders that outperform or match supervised and self‑supervised baselines on tasks such as monocular depth estimation, 3D object detection, 3D occupancy prediction, and end‑to‑end planning.

By Boshi Li, Xiaohui Wang, Xiaoyang Wu, Zhichao Li, Ya Yang, Naiyan Wang
arXiv Computer Vision
Sep 30

ORMA: Optimization-based Monocular 4D Reconstruction of Articulated Animals

ORMA is a training‑free framework that reconstructs articulated 4D representations of animals from monocular videos by decoupling pose and shape. It uses predicted pose as a reference for optimization and generative 3D priors to refine shape, aligning the result with the SMAL+ parametric model. The method combines per‑frame pose estimates with globally consistent camera poses, and further refines the reconstruction using self‑supervised DINO correspondences and temporal consistency, achieving improved accuracy on the new PAW4D benchmark and diverse real‑world videos.

By Xuyi Hu, Francesco Palandra, Shangzhe Wu, Daniel Cremers, Riccardo Marin, Silvia Zuffi
arXiv Computer Vision
Oct 12

Neural Voxel Dynamics: Learning Volumetric Feature Advection for 3D Physics in V-JEPA Latent Space

Neural Voxel Dynamics introduces a self‑supervised framework that learns 3D latent dynamics from monocular video by unprojecting V‑JEPAs features into a voxel grid using depth priors. It then applies Volumetric Feature Advection, an action‑conditioned transition model, to predict the evolution of these latent features directly in 3D, without relying on privileged physics‑engine states. The method is evaluated on synthetic and real datasets, demonstrating predictive fidelity, 3D geometric consistency, and physical plausibility via Physics‑IQ metrics.

By Zican Wang, Niloy Mitra
arXiv Machine Learning
1d ago

BAL: Bidirectional Autoregression in Latent Space for Learning Human Movement Representations

BAL (Bidirectional Autoregression in Latent Space) is a self‑supervised framework that learns human movement representations by predicting future and past latent states, mirroring how humans anticipate actions. It decomposes predictions into global sequence‑level semantics and local context‑independent movement, ensuring non‑trivial training signals and preventing collapse. Experiments on large datasets show BAL outperforms baselines in motion captioning, forecasting, and interpolation tasks.

By Genki Kinoshita, Ko Nishino
arXiv Computer Vision
6d ago

OnlyAct: Extracting Action Representations from Foundation Models for World Models

OnlyAct proposes extracting action representations from a pretrained foundation model (V-JEPA 2) instead of learning actions from scratch. It uses a label‑free contrastive adaptation framework to isolate motion while discarding scene information, avoiding reconstruction bottlenecks. The method is evaluated on diverse benchmarks, showing more expressive action representations and better transfer of actions to new scenes.

By Runjia Li, Minghao Chen, Junyu Xie, Philip Torr, Andrea Vedaldi, Tomas Jakab
arXiv Computer Vision
Nov 6

DeepSTEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans

DeepSTEP (STEP) is a Transformer-based framework that jointly performs tracking and pose estimation for animals and humans, eliminating the need for per-frame detections or post hoc matching. It uses discriminative model weight prediction to propagate identity across frames and introduces Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) to directly predict anatomically meaningful keypoints from an initial bounding box. Evaluated on diverse species datasets, STEP achieves strong performance and greater efficiency compared to existing pipelines.

By Shashikant Verma, Harish Katti, Soumyaratna Debnath, Yamuna Swamy, Shanmuganathan Raman