arXiv AI
MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 25
Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agents
Qwen‑Planner‑Agent is a closed‑loop AI‑for‑AI framework that enables large language models to act as both developers and participants in building advanced AI systems. The framework integrates data production, model training, and deployment through a shared action‑feedback‑verification contract, employing AI‑for‑Data, AI‑for‑Training, and AI‑driven model‑harness co‑evolution. It achieves top performance on MobilePA‑Bench by improving tool use, memory, skills, and sub‑agent coordination, while also showing gains on non‑mobile benchmarks.
arXiv Computer Vision
Aug 31
Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
The paper introduces GMA, a new benchmark for evaluating general mobile assistants in realistic, challenging scenarios. GMA expands on existing benchmarks by offering seven open‑source applications across diverse domains and 300 tasks organized into four difficulty tiers, ranging from simple actions to complex multi‑step workflows. The authors evaluate eight state‑of‑the‑art models, showing that performance drops sharply with task complexity, and conduct ablation studies on harness design—such as context retention and state tracking—to demonstrate how these choices can improve outcomes, especially for demanding workflows.
arXiv AI
Jun 19
ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis
arXiv:2605. 25160v2 Announce Type: replace Abstract: GUI agents powered by large language models are advancing rapidly, creating urgent needs for evaluation and training based on realistic environments.
arXiv AI
Jul 16
DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments
arXiv:2607. 13465v1 Announce Type: cross Abstract: LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes.
arXiv AI
Jul 15
PalmClaw: A Native On-Device Agent Framework for Mobile Phones
arXiv:2607. 13027v1 Announce Type: cross Abstract: Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action.
arXiv AI
Jun 2
MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents
arXiv:2512. 12634v4 Announce Type: replace Abstract: Mobile GUI Agents, AI agents capable of interacting with mobile applications on behalf of users, have the potential to transform human computer interaction.