arXiv Computer Vision

TrackFish3D: Self-Supervised 3D Tracking of Schooling Fish from Multi-view Videos

arXiv Computer Vision
Nov 6

BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting

BEAST3D is a self‑supervised framework that learns 3D visual representations from unlabeled, calibrated multi‑view video of animals. It uses a vision transformer to predict 3D Gaussian splats, reconstructing held‑out views via differentiable rendering while simultaneously segmenting the animal. The method works with as few as four views, produces viewpoint‑invariant features, and transfers to tasks such as novel view synthesis, multi‑view pose estimation, and neural encoding across four species.

By Yanchen Wang, Lenny Aharon, Wangshu Zhu, Kyle Daruwalla, Linghua Zhang, Jiaru Zou, Selmaan Chettih, Helen Hou, Liam Paninski, Matthew R Whiteway
arXiv Computer Vision
Sep 30

Sparse-View Interpretable 3D Animal Behavior Representations for Neural Encoding and Decoding

arXiv:2609.36217v1 Announce Type: new Abstract: A deeper understanding of brain function requires a precise, structured characterization of behavior.Yet, extracting behavioral representations from vi...

By Xinming Dai, Qihang Jin, Tianshu Tan, Baiyuan Chen, Hanrui Lyu, Lenny Aharon, Kyle Daruwalla, Xun Helen Hou, Matthew R. Whiteway, Liam Paninski, Yizi Zhang
arXiv Computer Vision
Nov 2

ReLift: Dense 3D Tracking as Lifted Trajectory Repair

ReLift presents a feed‑forward framework that treats dense 3D tracking as a problem of repairing and completing lifted trajectories. By alternating attention along and across trajectories, it corrects depth and correspondence errors, and fills in missing motion, all while training solely on simulated 3D trajectory data. The method improves lifted trajectories from any 2D tracker or geometry model, achieving state‑of‑the‑art accuracy on WorldTrack and running faster than leading 4D trackers.

By Assaf Singer, Maroun Khriesh, Noam Rotstein, Yoni Kasten, Or Litany
arXiv Computer Vision
Sep 7

Object Concepts Emerge from Motion

The paper introduces a biologically inspired framework that learns object‑centric visual representations from raw videos without human annotations or camera calibration. By using motion boundaries detected via optical flow and clustering to create pseudo‑instance masks, the method supervises a single‑image encoder with pixel‑level pairwise metric learning. Training on 195 million pseudo‑labeled frames and expanding to 421 million frames through Motion‑Verified Self‑Training, the approach yields Swin‑based encoders that outperform or match supervised and self‑supervised baselines on tasks such as monocular depth estimation, 3D object detection, 3D occupancy prediction, and end‑to‑end planning.

By Boshi Li, Xiaohui Wang, Xiaoyang Wu, Zhichao Li, Ya Yang, Naiyan Wang
arXiv Computer Vision
Oct 5

MoSE3: Learning World-Space SE(3) at Every Pixel

MoSE3 is a feed‑forward model that predicts dense SE(3) motion—full 6‑DoF rigid transforms—at every pixel from monocular RGB video, providing rotation, translation, and grouping information simultaneously. It achieves this by jointly learning 3D point tracks and rigidity embeddings, then differentiably fitting transforms within soft rigid clusters. The authors also release Art‑Kubric, a large synthetic dataset with dense SE(3) and rigidity labels, and demonstrate state‑of‑the‑art performance on both rigid and articulated benchmarks, with strong generalization to real‑world videos.

By Jiahuan Cheng, Zhiyi Li, Tian Xia, Ruojin Cai, Yilun Du, Qianqian Wang
arXiv Computer Vision
Sep 3

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views

TAPVid-MV is a new benchmark for tracking any point in 3D across multiple synchronized camera views. It comprises 284 sequences, 1,142 calibrated camera streams, and 109,769 point tracks, covering indoor and outdoor domains and derived from various modalities such as depth, LiDAR, SLAM, and simulation. The dataset is visually verified, and evaluation shows that current multi‑view trackers do not consistently outperform monocular trackers, highlighting geometry recovery as a key bottleneck.

By Skanda Koppula, Frano Rajic, Abdullah Faiz Ur Rahman, Yi Yang, Ignacio Rocco, Jeet Thakwani, Rishabh Kabra, Andrew Zisserman, Joao Carreira, Siyu Tang, Carl Doersch, Gabriel Brostow
arXiv Computer Vision
Aug 24

WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition

arXiv:2608.21281v1 Announce Type: new Abstract: Recent advances in field technology have led to a massive influx of in-the-wild video data for ecological science. The primary bottleneck in leveraging...

By Abigail G. Grassick, Jerome Tze-Hou Hsu, Ethan Lin, Ziang Liu, Max Whitton, Madelyn Hair, Liam Gutierrez, Haozheng Yu, Kristin Branson, Vivek Jayaraman, Michael A. Gil, Andrew M. Hein, Jennifer J. Sun
Hugging Face Trending Papers
Sep 28

VastMAT: A Large-Scale Multi-Category Benchmark for Multi-Animal Tracking

VastMAT is a large‑scale multi‑animal tracking benchmark featuring 2,947 videos, 337 animal categories, and over 3.6 million bounding boxes with 22,883 identity trajectories. It emphasizes high‑quality, expert‑reviewed annotations and introduces Seen‑category and Unseen‑category evaluation protocols, revealing significant challenges in tracking unseen animals. The authors also propose a lightweight Center‑Distance‑Augmented Association module that boosts HOTA scores for existing MOT methods without extra training.