arXiv:2608. 11434v1 Announce Type: new Abstract: Mobile agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile agent trajectories remains largely unexamined.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
arXiv:2512. 12634v4 Announce Type: replace Abstract: Mobile GUI Agents, AI agents capable of interacting with mobile applications on behalf of users, have the potential to transform human computer interaction.
arXiv:2605. 29486v2 Announce Type: replace-cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale.
arXiv:2608. 07775v1 Announce Type: new Abstract: Mobile agents have achieved promising results on clean online benchmarks such as AndroidWorld, yet their performance often degrades sharply in real-world deployment due to environmental variations and imperfect interface conditions.
The paper introduces CRATE, a two‑stage vision‑language model framework that evaluates mobile agents by reasoning about each step’s consequences and aggregating this evidence to assess task completion. It also presents CRATE‑S, an extension that evaluates operational safety. Experiments show CRATE and CRATE‑S outperform existing benchmarks, achieving high F1‑scores on AndroidWorld and MobileRisk datasets.
arXiv:2608. 05573v1 Announce Type: new Abstract: LLM agents increasingly execute long-horizon tasks through tool use and environment interaction, shifting evaluation from final-response scoring to verification of complete executions.
APPSim-Bench is a new benchmark for mobile GUI agents that uses controllable simulated apps to balance realism and reproducibility. It includes 557 tasks across 17 popular Chinese and English apps, with a coding-agent-assisted and human-verified workflow that ensures deterministic evaluation. Evaluation of 19 agents shows that autonomous mobile execution is still far from perfect, with the best model completing only 50.27% of tasks and many failures in longer workflows and numerical reasoning.