arXiv AI
Aug 5
UniNav: A Unified World-Action Diffusion Model for Visual Navigation
arXiv:2608. 03244v1 Announce Type: new Abstract: Image-goal visual navigation is a fundamental capability for embodied agents.
Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling
The paper introduces GenSight, a unified embodied world model that combines future video generation and dense perception into a single pretrained video diffusion transformer. GenSight employs a shared spatiotemporal representation for video, depth, embodiment segmentation, and first‑frame‑relative 2D flow, and uses lightweight task modulation to predict all modalities within the same architecture. The model supports both multi‑step future video generation with aligned dense predictions and single‑step dense perception from an observed video, demonstrating strong performance on embodied manipulation data across multiple robot embodiments and human‑hand interactions.
Related stories
arXiv Computer Vision
Oct 5
UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation
UniDynamics is a diffusion-based framework that generates future 4D dynamic scenes—comprising RGB, depth, and optical flow—from a single event-RGB pair, without needing long histories or control priors. It introduces an Event Latent Enhancement module to align event data into conditioning features and a Perceptual Dynamics Space within a multi-scale U‑Net to decouple and adaptively interact depth and flow, enforcing geometric and motion constraints for coherent predictions. Experiments on VKitti2 and DSEC show state‑of‑the‑art performance, producing high‑quality, temporally coherent, and 4D‑consistent predictions even under high‑speed motion blur.
arXiv Computer Vision
Oct 16
Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation
The paper introduces P³-WAM, a unified world action model for egocentric dexterous manipulation that integrates 3D perception, action-conditioned prediction, and goal-directed planning. It operates in three modes—Perceive, Predict, and Plan—using a mixture-of-transformers architecture that shares base weights and adapts to modality-specific inputs. Experiments demonstrate that unified training improves performance across all three capabilities, outperforming specialized baselines on 3D hand reconstruction and action-conditioned video prediction while producing more instruction‑adherent manipulation videos.
arXiv AI
Oct 9
World-Ego Modeling for Embodied Video Generation in Long-Horizon Navigation-Manipulation Tasks
The paper introduces the World-Ego Model (WEM), which separates an embodied video into a background‐only world component and an ego component comprising the robot and manipulated objects. By using role‑conditioned attention and asymmetric query budgets for state prediction, coupled with a semantic‑routed mixture‑of‑experts diffusion generator, WEM disentangles the generation of scene evolution and instruction following. The authors also release HTEWorld, a large dataset of 125K training clips and 300 evaluation trajectories for long‑horizon navigation‑manipulation tasks, and demonstrate that WEM outperforms prior methods on this benchmark.
arXiv Computer Vision
Sep 10
GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusion
arXiv:2605.12957v2 Announce Type: replace Abstract: Recent developments in generative models and large-scale datasets have substantially advanced 3D world generation, facilitating a broad range of do...
Hugging Face Trending Papers
Aug 4
UniNav: A Unified World-Action Diffusion Model for Visual Navigation
Image-goal visual navigation is a fundamental capability for embodied agents. Existing navigation policies efficiently predict waypoint trajectories but lack visual foresight, while navigation world models can anticipate future observations but often require costly planning rollouts.
arXiv AI
Oct 31
WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
WorldFlow is a generative world model that jointly predicts future video frames and query-based 3D trajectories using a unified Mixture-of-Transformers backbone. It represents 3D motion as continuous displacement tokens aligned with the video grid and optimizes a rectified-flow objective to couple explicit geometric dynamics with visual evolution. Trained on 1.44 million heterogeneous samples and evaluated on WorldFlowBench, it achieves the lowest average and final displacement errors among learning-based methods, notably outperforming baselines on spatial reasoning tasks.
arXiv Computer Vision
Sep 3
Spatially Aware World Action Model via Geometric Latent Diffusion
The paper introduces Spatially Aware World Action Model (SA‑WAM), a diffusion‑based framework that extends existing World Action Models by incorporating depth information alongside RGB to enable 3‑D‑aware action and future‑state prediction. SA‑WAM repurposes a pretrained video diffusion model, using a nonlinear encoding to map unbounded depth into the tokenizer’s bounded domain, thus preserving pretrained visual priors without 3‑D‑specific fine‑tuning. The model achieves state‑of‑the‑art performance on RoboCasa and LIBERO‑Plus benchmarks and demonstrates superior real‑world performance on a UR5 robotic arm in randomized environments, while also providing analysis linking world‑model prediction quality to rollout success.
arXiv Machine Learning
Jul 9
Gen4U: Unifying Video Generation and Understanding via Diffusion
arXiv:2607. 06856v1 Announce Type: cross Abstract: Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics.
Hugging Face Trending Papers
Oct 15
Gen4P: Video Generators are 4D Perceivers
Gen4P is a unified framework that transforms a pretrained video generator into a feed‑forward 4D perceiver, enabling recovery of 4D geometry and motion from monocular video without iterative denoising. It processes clean video latents in a single encoder pass and uses a lightweight cross‑attention decoder to predict requested 4D attributes, supporting flexible spatial sampling and continuous‑time queries for temporal flexibility. By incorporating a frozen pretrained video‑depth LoRA as a geometry prior, Gen4P improves performance on 3D point tracking, video depth, and camera pose estimation, achieving state‑of‑the‑art or competitive results on these tasks and demonstrating robust tracking of complex motion in in‑the‑wild videos.
arXiv Computer Vision
Sep 22
AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
arXiv:2609.22332v1 Announce Type: cross Abstract: Generalizable robot manipulation requires predicting how a scene will evolve, identifying where interactions are feasible, and determining how to act...
arXiv AI
Oct 15
Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence
arXiv:2610.18447v2 Announce Type: new Abstract: World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the fu...