CUA-Universe is a scalable environment-to-data pipeline that transforms real desktop software into hybrid GUI+CLI environments, enabling agents to coordinate visual inspection with command-line operations. It includes App-Forge for reproducible VMs and CLI surfaces, Task-Weave for generating diverse hybrid tasks, and Path-Steer for efficient rollouts and trajectory harvesting. Training on this data improves agent success and efficiency across multiple benchmarks, demonstrating the value of hybrid interaction.
arXiv:2609.38008v1 Announce Type: new Abstract: Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical us...
Computer use agents (CUAs) have demonstrated strong capabilities in completing digital tasks. However, existing CUAs either rely solely on graphical user interface (GUI) interactions, which are often...
arXiv:2606. 09426v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly operate in runtimes that combine visual desktop control, command-line execution, code editing, browsers, and external tools.
arXiv:2606. 24551v1 Announce Type: new Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions.
arXiv:2609.22000v1 Announce Type: new Abstract: Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Re...
OS-Helix is an end‑to‑end multimodal agent that unifies graphical user interface (GUI) actions and code execution into a single action space. It is trained on high‑quality hybrid data generated through application‑artifact composition and a decoupled instruction/evaluator pipeline, and further refined with online reinforcement learning using a step‑balanced GSPO algorithm. The resulting models achieve significant performance gains over hybrid baselines on the OSWorld‑Verified benchmark and demonstrate cross‑platform transfer to WindowsAgentArena.
OpenGCUA is an open‑stack project that aims to make generalist computer‑use agents (GCUs) fully accessible to the research community. The first release focuses on code execution, allowing agents to use intermediate results and programming skills directly in computer interaction, and introduces harness‑to‑model decision transfer to shift observation and task‑state decisions from fixed rules to the model. On the OSWorld‑v2 benchmark, the full configuration boosts task scores for all eight evaluated models—including GPT‑6 Astra and Claude Opus 5—by an average of +19.6 partial‑score points and cuts steps per task by 53 % compared to batched GUI‑only execution, while also providing a real‑device harness and cross‑device SDK. whyItMatters":"By releasing the full stack, OpenGCUA enables systematic study and broader experimentation with generalist computer‑use agents, addressing the current limitation of closed implementations and demonstrating significant performance gains across diverse models."
arXiv:2608. 02352v1 Announce Type: new Abstract: Native computer use offers a general interface for agents to operate almost any software available to people, but requires long-horizon state tracking, large-scale interactive experience, and learning from sparse yet verifiable outcomes.
arXiv:2610.15765v1 Announce Type: cross Abstract: Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were design...
arXiv:2605. 25160v2 Announce Type: replace Abstract: GUI agents powered by large language models are advancing rapidly, creating urgent needs for evaluation and training based on realistic environments.
arXiv:2609.40284v1 Announce Type: cross Abstract: Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on...
ToolCUA is an end‑to‑end agent that learns to optimally orchestrate GUI actions and tool calls for Computer Use Agents. It introduces a staged training pipeline that first scales interleaved GUI‑tool trajectories from static GUI data, then bootstraps decision making with single‑turn reinforcement learning, and finally refines performance via online agentic RL guided by a tool‑efficient path reward. On the OSWorld‑MCP benchmark, ToolCUA achieves 46.85% accuracy, a 66% relative improvement over the baseline and a 3.9% gain over GUI‑only models, setting a new state of the art for comparable‑scale models.