The paper introduces SolveBind, a GUI‑agent framework that tackles verification barriers such as CAPTCHAs by combining hierarchical solving (Perceive, Ground, Solve, Act, Verify) with a runtime supervisor that grounds actions in evidence and manages recovery budgets. Across nine CAPTCHA families and multiple backbone models, SolveBind outperforms baselines by 13.3–34.5 percentage points, achieving the highest success rates on all tested AndroidWorld backbones and improving dynamic verification performance compared to MobileAgent. The study also quantifies the trade‑offs between evidence‑grounded execution control, success rates, latency, and token usage, highlighting the impact of binding and recovery mechanisms on overall efficiency.
arXiv:2606. 02449v1 Announce Type: new Abstract: Multimodal agents are increasingly expected to operate interfaces on behalf of users, raising a central deployment question: can they truly substitute for humans in workflows that services deliberately protect against automation?
arXiv:2603. 23559v2 Announce Type: replace-cross Abstract: GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices.
AgentHorizon is a benchmark comprising 1,373 computer‑use tasks with instruction‑trajectory pairs collected from 166 hours of human‑recorded activity across three operating systems. The benchmark uses a paired design that swaps instructions to create negative tasks, testing judges’ ability to distinguish successful trajectories from those that completed a similar but incompatible request. Eleven judges were evaluated, with GPT‑5.5 achieving the highest balanced accuracy of 80.9% on the full trajectory subset, while tool‑use benefits some models but harms others, revealing large variability in judges’ performance.
VLAA-GUI is a modular framework for autonomous GUI agents that addresses early stopping and repetitive loops by integrating three core components: a Completeness Verifier, a Loop Breaker, and an on-demand Search Agent. The framework also includes optional Coding and Grounding Agents for specialized tasks. Evaluations on five backbones across Linux and Windows benchmarks show strong performance, with some models surpassing human results and the Loop Breaker significantly reducing wasted steps.
As GUI agents are increasingly deployed for mobile task automation, their reliability depends not only on final task outcomes, but also on whether they reach these outcomes through stable interaction...
CaptchaAgent is a structured visual reasoning framework designed to solve modern reasoning CAPTCHAs. It constructs a visual intermediate representation that records entities, attributes, and relations, then refines ambiguous regions and verifies counterfactual consistency before generating actions. The approach yields significant performance gains over baseline browser agents on Open CaptchaWorld and the Oedipus benchmark, highlighting key challenges such as exact counting, geometry, and ordered execution.
APPSim-Bench is a new benchmark for mobile GUI agents that uses controllable simulated apps to balance realism and reproducibility. It includes 557 tasks across 17 popular Chinese and English apps, with a coding-agent-assisted and human-verified workflow that ensures deterministic evaluation. Evaluation of 19 agents shows that autonomous mobile execution is still far from perfect, with the best model completing only 50.27% of tasks and many failures in longer workflows and numerical reasoning.
arXiv:2607. 25904v1 Announce Type: new Abstract: Graphical user interface task evaluation aims to determine whether a GUI agent has successfully completed a user instruction.
arXiv:2605. 25160v2 Announce Type: replace Abstract: GUI agents powered by large language models are advancing rapidly, creating urgent needs for evaluation and training based on realistic environments.
The paper introduces CRATE, a two‑stage vision‑language model framework that evaluates mobile agents by reasoning about each step’s consequences and aggregating this evidence to assess task completion. It also presents CRATE‑S, an extension that evaluates operational safety. Experiments show CRATE and CRATE‑S outperform existing benchmarks, achieving high F1‑scores on AndroidWorld and MobileRisk datasets.
arXiv:2509.23263v3 Announce Type: replace Abstract: Long-horizon GUI automation remains challenging due to error accumulation over extended interaction sequences. Process Reward Models (PRMs) provide...
arXiv:2607. 13027v1 Announce Type: cross Abstract: Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action.