Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottlenecked by two fundamental issues: current data s...
arXiv:2606. 24515v1 Announce Type: new Abstract: Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces.
Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions.
arXiv:2608. 05587v1 Announce Type: new Abstract: Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution.
MAI-Gym is an open, all-in-one platform that unifies GUI agent evaluation, data collection, and online reinforcement learning training across mobile, computer, and web environments. It supports ten established benchmarks, scales to 20,000 parallel environment instances, and enables automated synthesis of diverse GUI applications and tasks with execution-validated verifiers. Using this platform, the authors trained a 9B-parameter agent, MAI-Gym-9B, achieving state‑of‑the‑art performance on MobileWorld and OSWorld-Verified, and demonstrated that training on a mocked train‑ticketing app improves real‑world task success from 28.8% to 48.1% on a physical phone.
arXiv:2606. 12817v2 Announce Type: replace Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.
Jev-Mobile introduces a new approach for mobile GUI agents by separating high‑frequency lightweight execution from low‑frequency vision‑language model (VLM) planning. The VLM sets local goals, the accessibility tree provides a structured action space, and Jev, a fast typed decision model, repeatedly selects actions within this space, allowing multiple GUI actions per VLM decision. On the AndroidWorld task suite, Jev-Mobile achieves 79% task success, reduces mean end‑to‑end execution time by 32.7%, and cuts model API cost by 73.4% compared to a step‑wise VLM baseline.
arXiv:2609.22083v1 Announce Type: new Abstract: We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual too...
arXiv:2606. 31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation.
arXiv:2606. 12817v1 Announce Type: new Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.
The paper introduces GMA, a new benchmark for evaluating general mobile assistants in realistic, challenging scenarios. GMA expands on existing benchmarks by offering seven open‑source applications across diverse domains and 300 tasks organized into four difficulty tiers, ranging from simple actions to complex multi‑step workflows. The authors evaluate eight state‑of‑the‑art models, showing that performance drops sharply with task complexity, and conduct ablation studies on harness design—such as context retention and state tracking—to demonstrate how these choices can improve outcomes, especially for demanding workflows.
The paper introduces AnTrap, a benchmark that injects dynamic perturbations into Android GUI agent execution to evaluate robustness against runtime anomalies. It presents a taxonomy of anomalies across four layers—State, Thinking, Action, and Round—with ten subcategories, and a pipeline that maintains task solvability while adding realistic adversarial conditions. Experiments on 16 leading GUI models show universal vulnerability, and reinforcement learning can mitigate some traps but not deep contextual ones like state deadlock.