arXiv Computation and Language

iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents

arXiv Machine Learning

Jun 9

iOSWorld: A Benchmark for Personally Intelligent Phone Agents

arXiv:2606. 09764v1 Announce Type: new Abstract: A useful phone agent needs to be personally intelligent.

By Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov

agents benchmarks

arXiv AI

Jul 24

AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use

arXiv:2607. 20536v1 Announce Type: new Abstract: Tool-use agents that address day-to-day digital tasks such as ordering groceries must not only operate applications, but also interact with the user, e.

By Junzhi Chen, Harsh Trivedi, Jane Pan, Michael JQ Zhang, Tejas Srinivasan, Niranjan Balasubramanian, Ashish Sabharwal

llms agents benchmarks

arXiv Computer Vision

Aug 31

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

The paper introduces GMA, a new benchmark for evaluating general mobile assistants in realistic, challenging scenarios. GMA expands on existing benchmarks by offering seven open‑source applications across diverse domains and 300 tasks organized into four difficulty tiers, ranging from simple actions to complex multi‑step workflows. The authors evaluate eight state‑of‑the‑art models, showing that performance drops sharply with task complexity, and conduct ablation studies on harness design—such as context retention and state tracking—to demonstrate how these choices can improve outcomes, especially for demanding workflows.

By Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu

agents multimodal benchmarks

arXiv AI

Jul 15

PalmClaw: A Native On-Device Agent Framework for Mobile Phones

arXiv:2607. 13027v1 Announce Type: cross Abstract: Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action.

By Hongru Cai, Yongqi Li, Ran Wei, Wenjie Li

llms agents

arXiv AI

Aug 17

PhoneWorld: Scaling Phone-Use Agent Environments

arXiv:2605. 29486v2 Announce Type: replace-cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale.

By Yuxuan Liu, Xin Lai, Junyi Li, Pengyuan Lyu, Jason, Yiduo Guo, Zhengyao Fang, Yang Ding, Yi Zhang, Weinong Wang, Huawen Shen, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Chengquan Zhang, Han Hu, Benyou Wang, Ji-Rong Wen, Rui Yan, Zhengyang Tang

agents benchmarks

arXiv AI

Sep 10

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

APPSim-Bench is a new benchmark for mobile GUI agents that uses controllable simulated apps to balance realism and reproducibility. It includes 557 tasks across 17 popular Chinese and English apps, with a coding-agent-assisted and human-verified workflow that ensures deterministic evaluation. Evaluation of 19 agents shows that autonomous mobile execution is still far from perfect, with the best model completing only 50.27% of tasks and many failures in longer workflows and numerical reasoning.

By Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, Ziyang Wang, Yichen Gong

agents benchmarks

arXiv AI

Aug 12

SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information

arXiv:2608. 10692v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions.

By Junjie Ye, Zhuohui Sheng, Shaofan Liu, Yulun Zhu, Wenjie Fu, Dingwei Zhu, Ming Zhang, Yujiong Shen, Weichao Wang, Xin Zhao, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang, Pluto Zhou

llms benchmarks

Hugging Face Trending Papers

Sep 7

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these go...

agents benchmarks

arXiv AI

Aug 25

MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks

arXiv:2608.23035v1 Announce Type: new Abstract: As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capabili...

By Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu, Jiajun Liu, Sihan Cao, Long Chen, Weigao Sun, Feida Zhu, Yiran Zhong, Steven Hoi

llms agents reinforcement-learning robotics benchmarks

arXiv AI

Aug 7

Unified Agent: Managing Interactions across Devices

arXiv:2608. 05729v1 Announce Type: new Abstract: As capabilities rapidly increase, AI agents can move from running inside one app to acting across a user's devices over time.

By Xinshuang Liu, Runfa Blark Li, Shaoxiu Wei, Xin Lin, Truong Nguyen

llms agents multimodal benchmarks

arXiv AI

Aug 17

Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions

arXiv:2608. 14132v1 Announce Type: cross Abstract: Mobile GUI Agents powered by multimodal large language models (MLLMs) show promise in human-computer intelligence.

By Xiaokai Yan, Jingtao Ding, Yong Li, Zhiwen Yu

llms agents fine-tuning multimodal benchmarks

arXiv AI

Jun 2

MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents

arXiv:2512. 12634v4 Announce Type: replace Abstract: Mobile GUI Agents, AI agents capable of interacting with mobile applications on behalf of users, have the potential to transform human computer interaction.

By Youngmin Im, Byeongung Jo, Jaeyoung Wi, Seungwoo Baek, Tae Hoon Min, Joo Hyung Lee, Sangeun Oh, Insik Shin, Sunjae Lee

agents benchmarks

Related stories