Affora is a design system aimed at making software interfaces more readable by computer-use agents while still allowing designers visual freedom and maintaining familiar human workflows. The authors conducted three controlled studies on component implementations, visual variation, and interaction-design principles, using the results to create guidance from individual components to full sites, along with reusable implementations and executable checks. Evaluation on independently authored interfaces showed performance gains where Affora addressed existing deficits, with limited effects elsewhere, and a workflow case suggested reduced interaction cost.
arXiv:2608. 09944v1 Announce Type: cross Abstract: Modern web interfaces are increasingly difficult to use with screen readers, particularly when pages update dynamically or hide important structure behind visual layout.
arXiv:2607. 14443v1 Announce Type: new Abstract: Computer-use agents are becoming capable software operators, but their interface to desktop applications is still often a brittle motor layer: they look at screenshots, predict coordinates, click, and hope that the visible state changed as intended.
The study evaluates computer-use agents (CUAs) for blind users by conducting a three‑week diary study with eight participants using the OLLA prototype. Across 1,258 commands in 12 desktop applications, GPT‑5 achieved the highest success rate of 52.5%, while analysis uncovered failures in grounding, planning, constraint‑tracking, and termination. Interviews highlighted additional needs beyond automation for blind users.
arXiv:2510. 04452v3 Announce Type: replace-cross Abstract: Computer use agents (or "agents") are generative AI that automates actions within user interfaces from user commands.
arXiv:2604. 27996v3 Announce Type: replace Abstract: This paper examines how large language model (LLM) agents perform on scientific visualization (SciVis) tasks that require generating visualization workflows from natural-language instructions.
AgentLens is a mobile GUI agent that adapts its visual communication during task execution, offering Full UI, Partial UI, and GenUI modalities. It uses a Virtual Display to allow background operation while selectively overlaying visual information. In a study with 21 participants, 85.7% preferred AgentLens, which also scored highest on usability and adoption intent.
arXiv:2606. 26614v1 Announce Type: cross Abstract: Large language model (LLM) agents enable natural language interaction for scientific visualization (SciVis).
arXiv:2609.14512v1 Announce Type: new Abstract: Virtual worlds now host classrooms, meetings, conferences, shops, and social venues, and nearly every interaction they expose assumes a user who can sc...
arXiv:2608. 09278v1 Announce Type: cross Abstract: GUI agents have advanced rapidly, producing a growing body of frameworks, benchmarks, and applications.
Editable Visual Design introduces a new design paradigm that combines a Coding Agent with a Vision‑Language Model (VLM) and an image generation model. The VLM acts as the creative brain, understanding requirements, planning tasks, and judging aesthetics, while the image generator produces isolated visual assets on demand. The agent follows an "imagine first, then act" workflow, generating assets, writing native HTML/CSS, and refining the design through visual feedback, ultimately producing editable, layer‑wise artifacts with real text that can be adjusted via a graphical interface.
arXiv:2607. 19941v1 Announce Type: cross Abstract: As AI agents become integral to business workflows, establishing guiding user experience (UX) principles is crucial for ensuring user trust and successful adoption.
arXiv:2607. 15193v1 Announce Type: new Abstract: Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent.