arXiv:2609.25373v1 Announce Type: new
Abstract: On-device assistants require function-calling models that map natural language to local system actions, but existing resources emphasize web APIs or na...
By Ali Rezagholizadeh, Soheila Samiee
The paper introduces GMA, a new benchmark for evaluating general mobile assistants in realistic, challenging scenarios. GMA expands on existing benchmarks by offering seven open‑source applications across diverse domains and 300 tasks organized into four difficulty tiers, ranging from simple actions to complex multi‑step workflows. The authors evaluate eight state‑of‑the‑art models, showing that performance drops sharply with task complexity, and conduct ablation studies on harness design—such as context retention and state tracking—to demonstrate how these choices can improve outcomes, especially for demanding workflows.
By Yiqi Zhu, Feiyu Gao, Jiaxing Fan, Jiahui Zeng, Minggang Wu, Chenliang Li, Haiyang Xu, Peng Li, Ming Yan, Yang Liu
APPSim-Bench is a new benchmark for mobile GUI agents that uses controllable simulated apps to balance realism and reproducibility. It includes 557 tasks across 17 popular Chinese and English apps, with a coding-agent-assisted and human-verified workflow that ensures deterministic evaluation. Evaluation of 19 agents shows that autonomous mobile execution is still far from perfect, with the best model completing only 50.27% of tasks and many failures in longer workflows and numerical reasoning.
By Jintian Feng, Long Chen, Xiao Yu, Jiayi Dai, Chenglong Liu, Haoru Wang, Zizhen Xue, Yuxuan Shi, Ziyang Wang, Yichen Gong
arXiv:2606. 01099v1 Announce Type: cross Abstract: Command understanding systems in smart home ecosystems can automate device control and substantially improve user experience.
By Haowei Han, Kexin Hu, Weiwei Cai, Debiao Zhang, Bin Qin, Yuxiang Wang, Jiawei Jiang, Xiao Yan, Bo Du
Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these go...
arXiv:2607. 14661v1 Announce Type: new Abstract: Deploying large language models (LLMs) as personal assistants on mobile devices demands privacy, low latency, and offline availability, yet the computational cost of giant models clashes with strict edge-hardware budgets.
By Zhihan Jiang, Meng Li, Shenghao Liu, Keran Li, Ruiben Zhou, Xianjun Deng, Shuai Wang, Haipeng Dai