arXiv AI
Aug 7
StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
arXiv:2608. 05587v1 Announce Type: new Abstract: Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution.
Q-Reflector: Learning GUI Action Reflection with Generated Conditions
Q-Reflector is a standalone GUI action reflector built on Qwen3‑VL‑8B that learns to generate pre‑ and post‑conditions for each executed action and then evaluates execution success and task validity. It is trained with supervised fine‑tuning, teacher‑student distillation, and reward‑based refinement, and achieves 82.16% accuracy on 1,082 AndroidWorld transitions, outperforming baseline models. In MobileWorld tasks, it improves task success rates by 17.09 and 14.53 percentage points over no external reflection, and surpasses GPT‑5.2 and Claude Haiku 4.5 by 3.42 and 5.98 percentage points respectively.
Related stories
Hugging Face Trending Papers
Aug 6
StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions.
arXiv AI
Sep 1
GUI-PRA: Process Reward Agent for GUI Tasks
arXiv:2509.23263v3 Announce Type: replace Abstract: Long-horizon GUI automation remains challenging due to error accumulation over extended interaction sequences. Process Reward Models (PRMs) provide...
arXiv AI
Aug 26
Reflection with Action-Induced Visual Differences for Desktop GUI Agents
The paper introduces Evidence-First Reflection (EFR), a two-stage reflector for desktop GUI agents that separates visual difference extraction from outcome verification. EFR uses Set-of-Marks annotations to locate action sites and candidate changed regions, filters action-relevant changes, and then makes a final judgment based on cleaned evidence. Experiments on OSWorld-Verified and WindowsAgentArena show that EFR improves reflector accuracy by 7.11% and increases end-to-end task success by roughly 5–6%.
arXiv AI
Jul 1
Xiaomi-GUI-0 Technical Report
arXiv:2606. 31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation.
arXiv AI
Oct 9
AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks
AgentHorizon is a benchmark comprising 1,373 computer‑use tasks with instruction‑trajectory pairs collected from 166 hours of human‑recorded activity across three operating systems. The benchmark uses a paired design that swaps instructions to create negative tasks, testing judges’ ability to distinguish successful trajectories from those that completed a similar but incompatible request. Eleven judges were evaluated, with GPT‑5.5 achieving the highest balanced accuracy of 80.9% on the full trajectory subset, while tool‑use benefits some models but harms others, revealing large variability in judges’ performance.
arXiv AI
Aug 25
GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis
arXiv:2608.22847v1 Announce Type: new Abstract: Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottl...
arXiv AI
Jul 29
Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification
arXiv:2607. 25904v1 Announce Type: new Abstract: Graphical user interface task evaluation aims to determine whether a GUI agent has successfully completed a user instruction.
arXiv AI
Jul 29
Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
arXiv:2607. 26041v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks.
arXiv AI
Aug 18
UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations
arXiv:2608. 15930v1 Announce Type: new Abstract: Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution.
arXiv AI
Jul 20
SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction
arXiv:2607. 15550v1 Announce Type: new Abstract: Mobile graphical user interface (GUI) agents have demonstrated remarkable capabilities in automating complex tasks, yet they introduce critical safety risks where a single erroneous action can lead to irreversible consequences.
arXiv AI
Oct 16
MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments
MAI-Gym is an open, all-in-one platform that unifies GUI agent evaluation, data collection, and online reinforcement learning training across mobile, computer, and web environments. It supports ten established benchmarks, scales to 20,000 parallel environment instances, and enables automated synthesis of diverse GUI applications and tasks with execution-validated verifiers. Using this platform, the authors trained a 9B-parameter agent, MAI-Gym-9B, achieving state‑of‑the‑art performance on MobileWorld and OSWorld-Verified, and demonstrated that training on a mocked train‑ticketing app improves real‑world task success from 28.8% to 48.1% on a physical phone.