The paper introduces Richard, a prototype for voice‑first mobile interaction that separates voice sessions, task execution, and result delivery while linking them through persistent request records. It offers visual feedback via conversation and task views, and its backend handles immediate responses, dedicated services, and agent tasks. Evaluation on Android demonstrates controlled lifecycle verification and real‑world programming request execution, highlighting task continuity, user control, and service integration for intermittent user participation.
arXiv:2607. 13027v1 Announce Type: cross Abstract: Large Language Model (LLM) agents have moved beyond generating responses to executing multi-step tasks by calling tools, observing the results, and iteratively deciding the next action.
The paper introduces a framework that selectively reuses parts of previously executed mobile GUI control traces to reduce LLM inference overhead. It identifies six categories of variation that can affect trace reuse and shows that individual transitions can still be valid when checked against the current task context and UI state. Using graph-text templates, the system achieves roughly 80% fewer LLM tokens and up to 2‑fold speedups on Android and HarmonyOS devices.
AI agents are driving a new software paradigm, with the ability to autonomously call tools, extract information, manage memory, and complete tasks that span applications and data sources. Most existing end-user operating systems, however, are designed for application-centric workflows and offer little native support for AI agents.
arXiv:2609.25373v1 Announce Type: new Abstract: On-device assistants require function-calling models that map natural language to local system actions, but existing resources emphasize web APIs or na...
arXiv:2610.34735v1 Announce Type: new Abstract: An agent that truly knows and acts on behalf of its user has been a long-standing promise of AI assistants. Computer-use agents bring this promise with...
arXiv:2607. 15367v1 Announce Type: new Abstract: Desktop voice assistants are still dominated by cloud pipelines that ship raw audio off the machine and expose a fixed set of skills.
arXiv:2608. 05729v1 Announce Type: new Abstract: As capabilities rapidly increase, AI agents can move from running inside one app to acting across a user's devices over time.
JarvisGUI is a new benchmark that tests GUI agents on cross-device workflows involving Android, Windows, and Ubuntu, requiring transfer of intermediate results and coordination across heterogeneous platforms. It formulates tasks as input-output transformations under a lightweight type system, enabling automatic composition of multi-step, cross-device workflows and dynamic evaluation within a unified framework. The benchmark reveals that state-of-the-art open-source GUI agents struggle with state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management, exposing a critical capability gap invisible to existing benchmarks.
arXiv:2602. 15707v2 Announce Type: replace-cross Abstract: Real-time conversational assistants for procedural manual tasks often depend on video input, which can be computationally expensive and compromise user privacy.
The paper introduces GMA, a new benchmark for evaluating general mobile assistants in realistic, challenging scenarios. GMA expands on existing benchmarks by offering seven open‑source applications across diverse domains and 300 tasks organized into four difficulty tiers, ranging from simple actions to complex multi‑step workflows. The authors evaluate eight state‑of‑the‑art models, showing that performance drops sharply with task complexity, and conduct ablation studies on harness design—such as context retention and state tracking—to demonstrate how these choices can improve outcomes, especially for demanding workflows.
arXiv:2605. 29486v2 Announce Type: replace-cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale.
arXiv:2607. 13465v1 Announce Type: cross Abstract: LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes.