AndroidFlux is a benchmark designed to assess how well mobile-use multimodal agents recover from errors. It restores recorded trajectories to checkpoints before, during, and after recoverable errors, allowing agents to start from identical device states. A complementary reward‑modeling benchmark evaluates recovery‑action preferences based on downstream rollout outcomes, and an offline RL dataset is released to improve both error recovery and overall task success.
arXiv:2608. 07775v1 Announce Type: new Abstract: Mobile agents have achieved promising results on clean online benchmarks such as AndroidWorld, yet their performance often degrades sharply in real-world deployment due to environmental variations and imperfect interface conditions.
arXiv:2608.22847v1 Announce Type: new Abstract: Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottl...
Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottlenecked by two fundamental issues: current data s...
arXiv:2608. 11434v1 Announce Type: new Abstract: Mobile agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile agent trajectories remains largely unexamined.
The paper introduces an online reinforcement learning framework for UI navigation on live iOS apps, separating the agent into a Planning component and a UI Grounding model. It employs an LLM-as-a-Judge to supply dense step-level and terminal rewards, and uses PPO with a novel GiGPPO value estimation to stabilize training. Large-scale infrastructure enables concurrent virtual device training, and experiments on UINavBench, iOSWorld, and AndroidWorld show the approach outperforms supervised fine-tuning and terminal-only RL, achieving significant generalization gains and matching state‑of‑the‑art models on real iOS tasks.
arXiv:2606. 31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation.
The study evaluates the impact of integrating the Jev decision model into five AI agents across two application domains—embodied tasks in ALFWorld and mobile app use in AndroidWorld. While Jev consistently lowers decision cost and latency, the resulting task-level benefits differ by workflow: task success remains near baseline in ALFWorld but drops noticeably in two mobile agents. The authors find that auxiliary reasoning, additional interactions, failed recoveries, and premature completions can offset decision savings, leading to higher cost per successful outcome in some cases.
arXiv:2610.27954v1 Announce Type: new Abstract: Recent advances in large (vision) language models have enabled computer-use agents that automate digital workflows via fine-grained keyboard and mouse...
The paper introduces GMA, a new benchmark for evaluating general mobile assistants in realistic, challenging scenarios. GMA expands on existing benchmarks by offering seven open‑source applications across diverse domains and 300 tasks organized into four difficulty tiers, ranging from simple actions to complex multi‑step workflows. The authors evaluate eight state‑of‑the‑art models, showing that performance drops sharply with task complexity, and conduct ablation studies on harness design—such as context retention and state tracking—to demonstrate how these choices can improve outcomes, especially for demanding workflows.
Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions.
arXiv:2605. 29486v2 Announce Type: replace-cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale.
The paper introduces PAIR, a pipeline that creates user-conditioned application states to evaluate mobile GUI agents across different users, and RePAIR, a reinforcement learning method that incorporates personalization-aware interaction rewards. Experiments with six agents reveal significant variability in task success and subgoal achievement across users, especially when targeting personal content. RePAIR improves success rates over supervised fine-tuning, suggesting that learning from cross-user differences can enhance GUI-agent reliability.