arXiv:2608. 09944v1 Announce Type: cross Abstract: Modern web interfaces are increasingly difficult to use with screen readers, particularly when pages update dynamically or hide important structure behind visual layout.
By Santosh Patapati
Hybrid Computer Use (HCU) is a unified interface that enables general agents to interact with computers through three complementary channels: programmatic code and command-line tools, semantic accessibility (AX) features, and visual graphical user interfaces (GUI). The design emphasizes efficient AX interaction, batch execution, and AX diffs to reduce round trips, while a supplementary skill guides agents in selecting and coordinating these channels. On the OSWorld 2.0 benchmark, HCU outperforms other systems in both efficiency and task success, demonstrating the benefits of flexible, multi‑channel computer‑use interfaces.
By Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu
arXiv:2607. 14443v1 Announce Type: new Abstract: Computer-use agents are becoming capable software operators, but their interface to desktop applications is still often a brittle motor layer: they look at screenshots, predict coordinates, click, and hope that the visible state changed as intended.
By Yong Liu, Zhenyi Zhong, Zhanpeng Shi
The study evaluates computer-use agents (CUAs) for blind users by conducting a three‑week diary study with eight participants using the OLLA prototype. Across 1,258 commands in 12 desktop applications, GPT‑5 achieved the highest success rate of 52.5%, while analysis uncovered failures in grounding, planning, constraint‑tracking, and termination. Interviews highlighted additional needs beyond automation for blind users.
By Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan, Vikas Ashok
arXiv:2510. 04452v3 Announce Type: replace-cross Abstract: Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands.
By Jenny T. Liang, Titus Barik, Jeffrey Nichols, Eldon Schoop, Ruijia Cheng
arXiv:2610.15765v1 Announce Type: cross
Abstract: Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were design...
By Alan Wu, Wei Tao, Yiwen Guo