Hugging Face Trending Papers

StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions.

arXiv AI

Aug 7

StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

arXiv:2608. 05587v1 Announce Type: new Abstract: Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution.

By Linqiang Guo (Peter), Wei Liu (Peter), Li Gu (Peter), Yang Wang (Peter), Tse-Hsun (Peter), Chen

llms agents fine-tuning efficiency multimodal

arXiv AI

Jul 1

Xiaomi-GUI-0 Technical Report

arXiv:2606. 31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation.

By Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao, Niu Lian, Fazhan Liu, Hui Liu, Heng Qu, Qinzhuo Wu, Zhehao Yu, Tongbo Chen, Shiqi Cui, Anan Du, Shukai Jia, Yuanfa Li, Yike Liu, Wenchao Lu, Haoyuan Sun, Jiatong Sun, Cheng Tan, Yajie Wang, Changqiao Wu, Tao Xiong, Jiahui Yang, Yuxuan Yuan, Ruoceng Zhang, Shaojie Zhang, Jian Zhu, Jian Luan, Cong Zou

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Aug 11

UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents

arXiv:2607. 04425v2 Announce Type: replace-cross Abstract: Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction.

By Niu Lian, Tongbo Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Heng Qu, Shu-Tao Xia, Jinpeng Wang

agents efficiency multimodal

arXiv AI

Jun 12

Teach-and-Repeat: Accurately Extracting Operational Knowledge from Mobile Screen Demonstrations to Empower GUI Agents

arXiv:2606. 12817v1 Announce Type: new Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.

By Yudong Zhang (Honor Device Co., Ltd), Lei Hu (Honor Device Co., Ltd), Daoyang Liu (The Chinese University of Hong Kong, Hong Kong, China), Jiawei Liu (Honor Device Co., Ltd), Yangfan Luo (Honor Device Co., Ltd), Xingyu Liu (Honor Device Co., Ltd), Zuojian Wang (Honor Device Co., Ltd), Zhilin Gao (Honor Device Co., Ltd)

llms agents multimodal benchmarks

arXiv AI

Jun 15

GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models

arXiv:2606. 12817v2 Announce Type: replace Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.

By Yudong Zhang (Honor Device Co., Ltd), Lei Hu (Honor Device Co., Ltd), Daoyang Liu (The Chinese University of Hong Kong, Hong Kong, China), Jiawei Liu (Honor Device Co., Ltd), Yangfan Luo (Honor Device Co., Ltd), Zhilin Gao (Honor Device Co., Ltd), Zuojian Wang (Honor Device Co., Ltd)

llms agents multimodal benchmarks

arXiv AI

Jun 4

MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models

arXiv:2606. 04627v1 Announce Type: new Abstract: Mobile agents are increasingly expected to operate everyday applications from screenshots and language goals, where reliable control requires reasoning over screen affordances, multi-step navigation, and future state changes.

By Zhichao Yang, Yuanze Hu, Haojie Hao, Longkun Hao, Dongshuo Huang, Hongyu Lin, Gen Li, Lanqing Hong, Yihang Lou, Yan Bai

agents diffusion fine-tuning benchmarks

Hugging Face Trending Papers

Aug 12

CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

Mobile GUI agents remain brittle when deployed to applications absent from source training. We study novel-app generalization under a limited target interaction budget and without target demonstrations.

llms agents fine-tuning multimodal

arXiv AI

6d ago

CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

arXiv:2608. 11588v1 Announce Type: new Abstract: Mobile GUI agents remain brittle when deployed to applications absent from source training.

By Linqiang Guo (Peter), Li Gu (Peter), Zihuan Jiang (Peter), Zhixiang Chi (Peter), Siobhan Reid (Peter), Ziqiang Wang (Peter), Yuanhao Yu (Peter), Wei Liu (Peter), Yang Wang (Peter), Tse-Hsun (Peter), Chen

llms agents fine-tuning multimodal

arXiv AI

2d ago

PhoneWorld: Scaling Phone-Use Agent Environments

arXiv:2605. 29486v2 Announce Type: replace-cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale.

By Yuxuan Liu, Xin Lai, Junyi Li, Pengyuan Lyu, Jason, Yiduo Guo, Zhengyao Fang, Yang Ding, Yi Zhang, Weinong Wang, Huawen Shen, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Chengquan Zhang, Han Hu, Benyou Wang, Ji-Rong Wen, Rui Yan, Zhengyang Tang

agents benchmarks

Hugging Face Trending Papers

Jul 27

Scaling GUI Agents with Visual State Transitions

We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions).

agents fine-tuning multimodal benchmarks

Hugging Face Trending Papers

Jul 20

SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis

Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks. However, progress is limited by the lack of high-coverage, long-horizon interaction trajectories collected from element-rich and rapidly evolving apps.

llms agents fine-tuning multimodal

arXiv AI

Aug 7

AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents

arXiv:2608. 05891v1 Announce Type: new Abstract: Mobile GUI agents can operate apps through pixel perception and touch actions, making them a promising interface for collecting and improving long-horizon mobile interaction policies.

By Weikai Xu, Yunren Feng, Haoxiang Lei, Kun Huang, Yuxuan Liu, Kang Zhao, Xiaolin Hu, Shuo Shang, Bo An

agents reinforcement-learning fine-tuning benchmarks safety

Related stories