Affora is a design system aimed at making software interfaces more readable by computer-use agents while still allowing designers visual freedom and maintaining familiar human workflows. The authors conducted three controlled studies on component implementations, visual variation, and interaction-design principles, using the results to create guidance from individual components to full sites, along with reusable implementations and executable checks. Evaluation on independently authored interfaces showed performance gains where Affora addressed existing deficits, with limited effects elsewhere, and a workflow case suggested reduced interaction cost.
arXiv:2608. 09944v1 Announce Type: cross Abstract: Modern web interfaces are increasingly difficult to use with screen readers, particularly when pages update dynamically or hide important structure behind visual layout.
Hybrid Computer Use (HCU) is a unified interface that enables general agents to interact with computers through three complementary channels: programmatic code and command-line tools, semantic accessibility (AX) features, and visual graphical user interfaces (GUI). The design emphasizes efficient AX interaction, batch execution, and AX diffs to reduce round trips, while a supplementary skill guides agents in selecting and coordinating these channels. On the OSWorld 2.0 benchmark, HCU outperforms other systems in both efficiency and task success, demonstrating the benefits of flexible, multi‑channel computer‑use interfaces.
arXiv:2607. 14443v1 Announce Type: new Abstract: Computer-use agents are becoming capable software operators, but their interface to desktop applications is still often a brittle motor layer: they look at screenshots, predict coordinates, click, and hope that the visible state changed as intended.
The study evaluates computer-use agents (CUAs) for blind users by conducting a three‑week diary study with eight participants using the OLLA prototype. Across 1,258 commands in 12 desktop applications, GPT‑5 achieved the highest success rate of 52.5%, while analysis uncovered failures in grounding, planning, constraint‑tracking, and termination. Interviews highlighted additional needs beyond automation for blind users.
arXiv:2510. 04452v3 Announce Type: replace-cross Abstract: Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands.
arXiv:2610.15765v1 Announce Type: cross Abstract: Large language model agents are emerging as an important new class of users of desktop applications. However, existing desktop interfaces were design...
arXiv:2604. 27996v3 Announce Type: replace Abstract: This paper examines how large language model (LLM) agents perform on scientific visualization (SciVis) tasks that require generating visualization workflows from natural-language instructions.
AgentLens is a mobile GUI agent that adapts its visual communication during task execution, offering Full UI, Partial UI, and GenUI modalities. It uses a Virtual Display to allow background operation while selectively overlaying visual information. In a study with 21 participants, 85.7% preferred AgentLens, which also scored highest on usability and adoption intent.
The paper introduces MAGE, a framework that uses user manuals to enhance computer use agents (CUAs) for new software. MAGE retrieves relevant manual content during execution, generates procedural guidance, and adapts the agent policy through interaction. In training, it converts manuals into executable task-plan pairs to guide rollouts, reducing the need for human‑annotated demonstrations and improving task success rates across six applications.
arXiv:2606. 26614v1 Announce Type: cross Abstract: Large language model (LLM) agents enable natural language interaction for scientific visualization (SciVis).
arXiv:2609.14512v1 Announce Type: new Abstract: Virtual worlds now host classrooms, meetings, conferences, shops, and social venues, and nearly every interaction they expose assumes a user who can sc...
The article discusses how large language models can now generate user interfaces from natural‑language descriptions, and proposes a framework that turns HCI design knowledge into machine‑readable ‘skills’ that guide on‑demand UI construction. By treating the dialogue between user and AI as a shared cognitive workspace, these skills encode established HCI principles (e.g., Nielsen’s heuristics, WCAG criteria) as editable, version‑controlled artifacts. The authors outline a research agenda that envisions HCI moving from heuristic checklists to a machine‑executable, open‑source craft.