Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable.
arXiv:2607. 17790v1 Announce Type: cross Abstract: Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment.
MINT is a foundation model that directly predicts world-space two-hand trajectories from egocentric RGB video, jointly estimating camera motion, hand states, and hand presence in a single spatiotemporal representation. It uses an open-source labeling pipeline, EGOPIPELINE, to generate large-scale pseudo-labels for pretraining, followed by fine-tuning on a small set of high-quality joint annotations. The model outperforms existing multi-stage approaches in accuracy and speed, and generalizes zero‑shot to unseen egocentric datasets.
arXiv:2608.20308v2 Announce Type: replace Abstract: Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe ob...
arXiv:2608. 20308v1 Announce Type: new Abstract: Egocentric video offers scalable manipulation data for embodied AI, yet recovering metric 3D hand trajectories remains challenging due to severe object occlusion and frequent out-of-sight gaps.
Forge4D is a feed‑forward model that reconstructs temporally aligned 4D human representations from uncalibrated sparse‑view videos, enabling both novel view and novel time synthesis. It achieves this by jointly streaming 3D Gaussian reconstruction with dense motion prediction, using learnable state tokens for temporal consistency and a self‑supervised retargeting loss for motion prediction. Extensive experiments confirm its effectiveness on in‑domain and out‑of‑domain datasets.
arXiv:2608.29927v1 Announce Type: new Abstract: We address the problem of 3D body pose estimation of multiple interacting people from their egocentric views with centralized coordination. Each indivi...
arXiv:2609.08636v1 Announce Type: cross Abstract: Egocentric 4D interaction forecasting aims to anticipate both where future interactions will occur in 3D and how the human body will move to realize...
Ego-Exo4D is a large-scale dataset that offers synchronized egocentric and multi-view exocentric video for applications such as skill learning, procedural activity understanding, and embodied AI. The original dataset only includes sparse 3D human pose annotations, making dense motion reconstruction challenging. To address this, the authors introduce Ego-Exo4D-HM, a new dataset containing 4D human motion reconstructions for the Ego-Exo4D captures, along with a reconstruction pipeline and accompanying code and documentation.
arXiv:2605.14854v3 Announce Type: replace-cross Abstract: Human Mesh Recovery (HMR) is fundamentally ambiguous: under occlusion or weak depth cues, multiple 3D bodies can explain the same image evide...
arXiv:2610.20124v1 Announce Type: new Abstract: Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand...
arXiv:2604. 01001v2 Announce Type: replace-cross Abstract: We introduce EgoSim, a closed-loop egocentric world simulator that generates spatially consistent interaction videos and persistently updates the underlying 3D scene state for continuous simulation.