arXiv Computation and Language
iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Jun 9
iOSWorld: A Benchmark for Personally Intelligent Phone Agents
arXiv:2606. 09764v1 Announce Type: new Abstract: A useful phone agent needs to be personally intelligent.
arXiv AI
Jul 24
AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use
arXiv:2607. 20536v1 Announce Type: new Abstract: Tool-use agents that address day-to-day digital tasks such as ordering groceries must not only operate applications, but also interact with the user, e.
arXiv Computer Vision
Aug 31
Benchmarking General Mobile Assistants in Challenging Real-World Scenarios
The paper introduces GMA, a new benchmark for evaluating general mobile assistants in realistic, challenging scenarios. GMA expands on existing benchmarks by offering seven open‑source applications across diverse domains and 300 tasks organized into four difficulty tiers, ranging from simple actions to complex multi‑step workflows. The authors evaluate eight state‑of‑the‑art models, showing that performance drops sharply with task complexity, and conduct ablation studies on harness design—such as context retention and state tracking—to demonstrate how these choices can improve outcomes, especially for demanding workflows.
arXiv AI
Jul 15
PalmClaw: A Native On-Device Agent Framework for Mobile Phones
arXiv:2607. 13027v1 Announce Type: cross Abstract: Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action.
arXiv AI
Aug 17
PhoneWorld: Scaling Phone-Use Agent Environments
arXiv:2605. 29486v2 Announce Type: replace-cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale.
arXiv AI
Sep 10
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
APPSim-Bench is a new benchmark for mobile GUI agents that uses controllable simulated apps to balance realism and reproducibility. It includes 557 tasks across 17 popular Chinese and English apps, with a coding-agent-assisted and human-verified workflow that ensures deterministic evaluation. Evaluation of 19 agents shows that autonomous mobile execution is still far from perfect, with the best model completing only 50.27% of tasks and many failures in longer workflows and numerical reasoning.