← Back to all news
arXiv Computation and Language By Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu

Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

Read the original on arXiv Computation and Language →

Hybrid Computer Use (HCU) is a unified interface that enables general agents to interact with computers through three complementary channels: programmatic code and command-line tools, semantic accessibility (AX) features, and visual graphical user interfaces (GUI). The design emphasizes efficient AX interaction, batch execution, and AX diffs to reduce round trips, while a supplementary skill guides agents in selecting and coordinating these channels. On the OSWorld 2.0 benchmark, HCU outperforms other systems in both efficiency and task success, demonstrating the benefits of flexible, multi‑channel computer‑use interfaces.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Jun 24

GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

arXiv:2606. 24551v1 Announce Type: new Abstract: Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions.

By Xiao Zhou, Siyue Zhang, Yilun Zhao, Jinbiao Wei, Tingyu Song, Arman Cohan, Chen Zhao
agentsbenchmarks
More like this →
arXiv Computer Vision
Oct 16

OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

OpenGCUA is an open‑stack project that aims to make generalist computer‑use agents (GCUs) fully accessible to the research community. The first release focuses on code execution, allowing agents to use intermediate results and programming skills directly in computer interaction, and introduces harness‑to‑model decision transfer to shift observation and task‑state decisions from fixed rules to the model. On the OSWorld‑v2 benchmark, the full configuration boosts task scores for all eight evaluated models—including GPT‑6 Astra and Claude Opus 5—by an average of +19.6 partial‑score points and cuts steps per task by 53 % compared to batched GUI‑only execution, while also providing a real‑device harness and cross‑device SDK. whyItMatters":"By releasing the full stack, OpenGCUA enables systematic study and broader experimentation with generalist computer‑use agents, addressing the current limitation of closed implementations and demonstrating significant performance gains across diverse models."

By Hanzhang Zhou, Long Li, Quyu Kong, Lei Wang, Panrong Tong, Long Chen, Tianyu Xia, Gongjie Zhang, Xu Zhang, Chen Liu, Jianan Zhang, Chenglin Cai, Zhengwen Zeng, Steven Hoi, Yue Wang
llmsagentsbenchmarks
More like this →