arXiv AI By Songze Li, Xiaoke Guo, Tianqi Liu, Biao Yi, Zhaoyan Gong, Zhiqiang Liu, Huajun Chen, Wen Zhang

What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning

Read the original on arXiv AI →

arXiv:2604. 06995v2 Announce Type: replace Abstract: Existing Graphical User Interface (GUI) reasoning tasks remain challenging, particularly in UI understanding.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Sep 2

UI-Venus-2 Technical Report

UI‑Venus‑2 is a general‑purpose foundation GUI agent that operates across mobile, web, and desktop environments using a unified closed‑loop reasoning‑action framework. The report details how the system expands environment coverage to over 170 multilingual mobile apps and native desktop OSes, scales task generation through a deep‑research pipeline, and enhances verification with trace‑level and sample‑level evaluators that use visual keypoints and multi‑model voting. Safety‑aware mechanisms are also incorporated to control consequential actions, positioning UI‑Venus‑2 as an efficient, open‑source tool for more generalizable, verifiable, and self‑reflective agents in real‑world applications.

By Venus Team, Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Yusong Hu, Jianrong Jiang, Jianguo Li, Runze Li, Jinzhen Lin, Zhenyu Ma, Changhua Meng, Han Peng, Xinyu Qiu, Shuheng Shen, Zhongyi Shui, Weiqiang Wang, Ming Wen, Zhuoer Xu, Hang Yan, Kaiwen Yang, Ruilin Yao, Nanjun Yu, Zhengwen Zeng, Lianrui Zhang, Yunzhu Zhang, Zhe Zhao, Beitong Zhou
arXiv AI
Jun 15

GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models

arXiv:2606. 12817v2 Announce Type: replace Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.

By Yudong Zhang (Honor Device Co., Ltd), Lei Hu (Honor Device Co., Ltd), Daoyang Liu (The Chinese University of Hong Kong, Hong Kong, China), Jiawei Liu (Honor Device Co., Ltd), Yangfan Luo (Honor Device Co., Ltd), Zhilin Gao (Honor Device Co., Ltd), Zuojian Wang (Honor Device Co., Ltd)