The paper proposes ‘meta access’—a structured, permissioned view of an interface’s state—to enhance large language model chat UIs. It outlines a ladder of access levels, from feature tutoring to full orchestration, and demonstrates the concept with the Meta Chat prototype, which uses summaries, settings, and message links to enable reversible changes and a browsable tutorial library. Meta access aims to transform chat from a single conversation window into a control layer for the entire workspace while addressing risks such as grounding, privacy, and prompt injection.
By Julian Kunkel, Ali Doosthosseini
AgentLens is a mobile GUI agent that adapts its visual communication during task execution, offering Full UI, Partial UI, and GenUI modalities. It uses a Virtual Display to allow background operation while selectively overlaying visual information. In a study with 21 participants, 85.7% preferred AgentLens, which also scored highest on usability and adoption intent.
By Jeonghyeon Kim, Byeongjun Joung, Junwon Lee, Joohyung Lee, Taehoon Min, Sunjae Lee
The article discusses how large language models can now generate user interfaces from natural‑language descriptions, and proposes a framework that turns HCI design knowledge into machine‑readable ‘skills’ that guide on‑demand UI construction. By treating the dialogue between user and AI as a shared cognitive workspace, these skills encode established HCI principles (e.g., Nielsen’s heuristics, WCAG criteria) as editable, version‑controlled artifacts. The authors outline a research agenda that envisions HCI moving from heuristic checklists to a machine‑executable, open‑source craft.
By Nathan Conklin, Miranda Capra, Chris North
The paper explores how generative AI can lower the barrier to personalizing health dashboards by enabling users to co-design interfaces in Figma Make. In a study with 14 participants, redesigns of Google and Apple Health focused on personal context, future planning, and interactive experiences, though conversational AI designs tended toward chat-window conventions. AI facilitated the materialization of loosely articulated ideas, yet model defaults and generation latency influenced iteration, and the process highlighted interpretability and accountability over privacy, trust, and emotional safety.
By Karthik S. Bhat, Vidhi Shah, Vedika Agnihotri, Dong Whi Yoo, Koustuv Saha
arXiv:2601. 18305v2 Announce Type: replace Abstract: Despite numerous Graphical User Interface (GUI) agents claiming to automate user interaction tasks, to date, few achieve satisfactory interaction capability with human users in real-world scenarios.
By Xuan Wang, Siyuan Su, Quantong Fu, Yongxiang Hu, Yangfan Zhou
Hybrid Computer Use (HCU) is a unified interface that enables general agents to interact with computers through three complementary channels: programmatic code and command-line tools, semantic accessibility (AX) features, and visual graphical user interfaces (GUI). The design emphasizes efficient AX interaction, batch execution, and AX diffs to reduce round trips, while a supplementary skill guides agents in selecting and coordinating these channels. On the OSWorld 2.0 benchmark, HCU outperforms other systems in both efficiency and task success, demonstrating the benefits of flexible, multi‑channel computer‑use interfaces.
By Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu