arXiv AI
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computer Vision
Oct 8
Juno: Taming Predictive Latents for Vision-Language-Action Models
arXiv:2610.09940v1 Announce Type: cross Abstract: Joint-embedding predictive architectures (JEPAs) predict masked or future observations in representation space, offering a natural source of predicti...
arXiv AI
Sep 25
Jev-Mobile: Jev as an Executor for Mobile GUI Agents
Jev-Mobile introduces a new approach for mobile GUI agents by separating high‑frequency lightweight execution from low‑frequency vision‑language model (VLM) planning. The VLM sets local goals, the accessibility tree provides a structured action space, and Jev, a fast typed decision model, repeatedly selects actions within this space, allowing multiple GUI actions per VLM decision. On the AndroidWorld task suite, Jev-Mobile achieves 79% task success, reduces mean end‑to‑end execution time by 32.7%, and cuts model API cost by 73.4% compared to a step‑wise VLM baseline.
arXiv Machine Learning
Oct 7
Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?
The paper investigates how a small grounding model for a GUI agent should receive the action type. Using Qwen2‑VL‑2B fine‑tuned with LoRA on Android, the authors compare a flat baseline with five conditioning methods—auxiliary loss, hard‑routed action word, additive learned embedding, prepended learned token, and prompt word—under matched data, compute, and decoding settings. Results show that auxiliary loss, additive embedding, and prompt word each improve hit@0.10 by five to seven points over the baseline, while hard routing and prepended token perform similarly to the baseline; however, removing a problematic off‑screen touch point class lifts the baseline by nearly seven points, after which no method outperforms it. "whyItMatters":"The study identifies effective strategies for encoding action type in small grounding models, which could improve the accuracy of GUI agents in real‑world Android applications."
arXiv AI
Oct 5
Keeping JEPA World Models Plannable When Little of the Frame Moves
The paper introduces SLIM, a pushing benchmark that tests language‑based goal specification for latent world models. It shows that a vanilla LeWM fails almost entirely due to an action‑insensitive encoder, but adding an inverse‑dynamics auxiliary loss restores encoder sensitivity and dramatically improves success rates. With the repaired latent, a lightweight language‑goal head can plan from natural language sentences, achieving high performance on navigation and pushing tasks without retraining the world model.
arXiv Machine Learning
Oct 23
SWE-VERIFIER: Selective Verification for Cost-Efficient Coding Agents
SWE-VERIFIER is a lightweight generative model that selectively triggers candidate sampling and verification for coding agents, scoring proposed actions to decide when additional sampling is needed and ranking the resulting actions before execution. Trained on Qwen3.5-4B with preference data, it improves resolution on MiniMax-M2.7 from 75.47% to 77.20% at 1.09× cost and on DeepSeek-V4-Flash from 72.60% to 74.27% at 1.13× cost, while verifying fewer than 4% of steps. The approach demonstrates that selective sampling and lightweight verification can enhance coding‑agent performance and inference efficiency with minimal overhead.
arXiv AI
Sep 30
V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
arXiv:2609.37250v1 Announce Type: cross Abstract: World-action models (WAMs) couple future visual-state prediction with action generation. By adapting video generators or image-editing models pretrai...