← Back to all news
arXiv Computer Vision By Jiaming Hu, Shi Bo, Yan Zheng, Tian Wang, Florian Dubost, Alejandro Mottini, Junze Liu, Arvind Srinivasan, Kai Zhong, Kun Qian, Sharon Gao, Vianne Gao, Houyu Zhang, Qingjun Cui

SWIM: Learning What Matters from Behaviors for JEPA World Models

Read the original on arXiv Computer Vision →

SWIM (State Weighting with Inverse-Dynamics Model) learns which state features matter for joint-embedding predictive architectures (JEPAs) by training an inverse-dynamics model on offline trajectories to predict actions from states and future displacements. The resulting feature weights define a weighted state metric that aligns latent and state distances, improving state-supervised training without extra planning-time computation. Experiments on five simulated robotic manipulation tasks show that SWIM consistently outperforms uniform state weighting in planning success.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv AI
Oct 12

LeWAM: End-to-End World and Action Modeling with JEPAs

LeWAM is an end‑to‑end joint‑embedding predictive architecture that learns compact latent world models without pixel reconstruction and directly supports policy execution and planning. It trains jointly on next‑latent‑state prediction and action‑flow matching while imposing an isotropic Gaussian constraint on the latent state. The approach yields significant performance gains, raising visual‑planning success rates from 28.6 % to 89.7 % on contact‑rich tasks, from 10.9 % to 46.2 % on long‑horizon tasks, and speeding up planning by 32.3×, while remaining competitive on behavior‑cloning tasks and providing interpretable control‑relevant representations.

By Minghao Fu, Tavis Siebert, Eryk Halicki, Randall Balestriero
ragdiffusionroboticssafety
More like this →
arXiv AI
Jul 1

Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

arXiv:2606. 31232v1 Announce Type: new Abstract: Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations.

By Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan, Yujia Yang, Bingkang Shi, Tianyu Zong, Hongzhu Yi, Guoqing Chao, Xingchen Chen, Tiankun Yang, Chenxi Bao, Tao Yu, Jingjing Zhou, Jungang Xu
rag
More like this →
arXiv AI
Oct 19

Beyond Action Prediction: Learning Search-Free Planning in Latent Space

The paper introduces Hindsight Inverse JEPA Planning (HIJP), a search‑free framework that replaces costly online action sequence optimization with a goal‑conditioned inverse model. HIJP trains by relabeling future observations as goals, providing action supervision across multiple horizons and shaping visual representations. The method also aligns predicted actions with a latent predictor model to enforce action‑transition consistency, achieving high success rates and significant speedups on visual control tasks.

By Guanchen Du, Jinhao Du, Yanpeng Sun, Shan Zhang, Na Zhao, Zechao Li
diffusion
More like this →