Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

The paper introduces GMA, a new benchmark for evaluating general mobile assistants in realistic, challenging scenarios. GMA expands on existing benchmarks by offering seven open‑source applications across diverse domains and 300 tasks organized into four difficulty tiers, ranging from simple actions to complex multi‑step workflows. The authors evaluate eight state‑of‑the‑art models, showing that performance drops sharply with task complexity, and conduct ablation studies on harness design—such as context retention and state tracking—to demonstrate how these choices can improve outcomes, especially for demanding workflows.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

• agents
• multimodal
• benchmarks