arXiv:2607. 17790v1 Announce Type: cross Abstract: Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment.
Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable.
EgoGen is a synthetic data generator designed for egocentric perception tasks, producing accurate ground‑truth training data by simulating natural human movements that steer virtual cameras. Its core is a human motion synthesis model that uses egocentric visual inputs to sense 3D environments, combined with collision‑avoiding motion primitives and a two‑stage reinforcement learning approach for closed‑loop perception and movement. The system eliminates the need for pre‑defined global paths, works in dynamic settings, and supports tasks such as mapping, localization, camera tracking, and human mesh recovery from first‑person views.
arXiv:2603. 09731v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly considered as a foundation for embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.
MINT is a foundation model that directly predicts world-space two-hand trajectories from egocentric RGB video, jointly estimating camera motion, hand states, and hand presence in a single spatiotemporal representation. It uses an open-source labeling pipeline, EGOPIPELINE, to generate large-scale pseudo-labels for pretraining, followed by fine-tuning on a small set of high-quality joint annotations. The model outperforms existing multi-stage approaches in accuracy and speed, and generalizes zero‑shot to unseen egocentric datasets.
arXiv:2609.01276v1 Announce Type: new Abstract: Complete 3D perception from egocentric video requires recovering the surrounding scene and the wearer's full-body motion in a shared metric frame. Exis...
BinoGen is an automated framework that generates large-scale, embodiment-aware egocentric binocular visual experiences in indoor environments. It models environmental and observer variation through generative scene synthesis, probabilistic object instantiation, appearance randomization, stochastic trajectory generation, and configurable binocular camera setups, producing synchronized videos with dense multimodal supervision such as depth maps, optical flow, surface normals, semantic maps, object coordinates, and camera poses. Using BinoGen, the authors created a dataset of over 20 million annotated images, demonstrating that incorporating this data improves real-world visual perception tasks like depth estimation, object detection, and video object tracking, and that embodiment-specific adaptation enhances performance while joint training enables a single model to perform competitively across different observer embodiments.
arXiv:2611.08477v1 Announce Type: new Abstract: Modern video models possess strong visual priors, yet do not natively represent human action in metric 3D space. We present EgoGeneralist, which adapts...
VISTA (Visually Impaired Scene and Task Annotations) is a multimodal egocentric benchmark designed to evaluate goal‑oriented assistance for blind and low‑vision users. It comprises about 1,000 recordings from Project Aria glasses, each providing six synchronized sensor streams and three types of annotations across ten task categories, from object localization to hazard detection. The benchmark tests vision‑language models in zero‑shot settings and introduces a lightweight multimodal baseline that integrates all six modalities, revealing that current models can describe scenes but struggle to generate effective assistive guidance.
Modern video models possess strong visual priors, yet do not natively represent human action in metric 3D space. We present EgoGeneralist, which adapts pretrained video generators to unify action-cond...
SOMA is a new benchmark that tests fine‑grained egocentric body‑motion understanding, featuring 1,632 human‑verified questions across body configuration, motion dynamics, and temporal reasoning. The study evaluates 37 multimodal large language models (MLLMs) and finds that humans achieve 68.5% accuracy while the best MLLM (Qwen3.6‑27B) reaches only 50.0% zero‑shot accuracy. The authors also provide a training dataset and a multimodal baseline to help close the performance gap and highlight common failure modes such as misreading body parts and temporal errors.
arXiv:2607. 02417v1 Announce Type: cross Abstract: Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent.