TrackFish3D: Self-Supervised 3D Tracking of Schooling Fish from Multi-view Videos
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
The Flow has not summarised this story yet — read it at arXiv Computer Vision.
BEAST3D is a self‑supervised framework that learns 3D visual representations from unlabeled, calibrated multi‑view video of animals. It uses a vision transformer to predict 3D Gaussian splats, reconstructing held‑out views via differentiable rendering while simultaneously segmenting the animal. The method works with as few as four views, produces viewpoint‑invariant features, and transfers to tasks such as novel view synthesis, multi‑view pose estimation, and neural encoding across four species.
arXiv:2609.36217v1 Announce Type: new Abstract: A deeper understanding of brain function requires a precise, structured characterization of behavior.Yet, extracting behavioral representations from vi...
ReLift presents a feed‑forward framework that treats dense 3D tracking as a problem of repairing and completing lifted trajectories. By alternating attention along and across trajectories, it corrects depth and correspondence errors, and fills in missing motion, all while training solely on simulated 3D trajectory data. The method improves lifted trajectories from any 2D tracker or geometry model, achieving state‑of‑the‑art accuracy on WorldTrack and running faster than leading 4D trackers.
The paper introduces a biologically inspired framework that learns object‑centric visual representations from raw videos without human annotations or camera calibration. By using motion boundaries detected via optical flow and clustering to create pseudo‑instance masks, the method supervises a single‑image encoder with pixel‑level pairwise metric learning. Training on 195 million pseudo‑labeled frames and expanding to 421 million frames through Motion‑Verified Self‑Training, the approach yields Swin‑based encoders that outperform or match supervised and self‑supervised baselines on tasks such as monocular depth estimation, 3D object detection, 3D occupancy prediction, and end‑to‑end planning.
arXiv:2610.01314v1 Announce Type: new Abstract: Dynamic scenes may be captured by a moving camera, multiple video streams, or images taken at different times. These observations reveal complementary...
MoSE3 is a feed‑forward model that predicts dense SE(3) motion—full 6‑DoF rigid transforms—at every pixel from monocular RGB video, providing rotation, translation, and grouping information simultaneously. It achieves this by jointly learning 3D point tracks and rigidity embeddings, then differentiably fitting transforms within soft rigid clusters. The authors also release Art‑Kubric, a large synthetic dataset with dense SE(3) and rigidity labels, and demonstrate state‑of‑the‑art performance on both rigid and articulated benchmarks, with strong generalization to real‑world videos.