arXiv:2606. 09426v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools.
CUA-Universe is a scalable environment-to-data pipeline that transforms real desktop software into hybrid GUI+CLI environments, enabling agents to coordinate visual inspection with command-line operations. It includes App-Forge for reproducible VMs and CLI surfaces, Task-Weave for generating diverse hybrid tasks, and Path-Steer for efficient rollouts and trajectory harvesting. Training on this data improves agent success and efficiency across multiple benchmarks, demonstrating the value of hybrid interaction.
arXiv:2605. 25160v2 Announce Type: replace Abstract: GUI agents powered by large language models are advancing rapidly, creating urgent needs for evaluation and training based on realistic environments.
arXiv:2606. 24551v1 Announce Type: new Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions.
arXiv:2607. 05174v1 Announce Type: new Abstract: Language agents, i.
JarvisGUI is a new benchmark that tests GUI agents on cross-device workflows involving Android, Windows, and Ubuntu, requiring transfer of intermediate results and coordination across heterogeneous platforms. It formulates tasks as input-output transformations under a lightweight type system, enabling automatic composition of multi-step, cross-device workflows and dynamic evaluation within a unified framework. The benchmark reveals that state-of-the-art open-source GUI agents struggle with state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management, exposing a critical capability gap invisible to existing benchmarks.
arXiv:2609.22000v1 Announce Type: new Abstract: Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Re...
The paper introduces ALIGN, a lightweight framework that automatically generates enriched interfaces to reduce agent‑environment misalignment in large language model agents. By augmenting static environment information and step‑wise observations, ALIGN improves performance across diverse tasks such as embodied navigation, web browsing, and tool use, achieving up to a 45.67% success rate increase in ALFWorld. The method works without altering agent logic or environment code and generalizes across different agent architectures and LLM backbones.
ComponentBench is a new benchmark that evaluates computer‑use agents at the component level on modern web UIs. It contains 97 canonical UI components and 2,910 programmatically verified tasks, along with cleaned human reference trajectories for measuring task success and interaction efficiency. The benchmark also offers a scalable pipeline for auditing structural difficulty and synthesizing failure analyses across tasks and component families.
arXiv:2608. 09278v1 Announce Type: cross Abstract: GUI agents have advanced rapidly, producing a growing body of frameworks, benchmarks, and applications.
OS-Helix is an end‑to‑end multimodal agent that unifies graphical user interface (GUI) actions and code execution into a single action space. It is trained on high‑quality hybrid data generated through application‑artifact composition and a decoupled instruction/evaluator pipeline, and further refined with online reinforcement learning using a step‑balanced GSPO algorithm. The resulting models achieve significant performance gains over hybrid baselines on the OSWorld‑Verified benchmark and demonstrate cross‑platform transfer to WindowsAgentArena.
OpenClaw has emerged as a leading agent framework for complex task automation, yet it faces insufficient cross-platform GUI interaction support and a well-built self-evolution mechanism. These flaws limit its adaptation to diverse device ecosystems and prevent performance improvements through continuous learning from execution experience.