Hugging Face Trending Papers

Scaling GUI Agents with Visual State Transitions

We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions).

arXiv AI

Jul 28

Scaling GUI Agents with Visual State Transitions

arXiv:2607. 24112v1 Announce Type: new Abstract: We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents.

By Xiangyan Liu, Kaixin Li, Haonan Wang, Biao Wu, Meng Fang, Longxu Dou, Chao Du, Michael Qizhe Shieh, Tianyu Pang

agents fine-tuning multimodal benchmarks

arXiv AI

Jul 21

EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding

arXiv:2607. 17050v1 Announce Type: cross Abstract: GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery.

By Yaohan Yang, Minglei Shi, Borui Zhang, Jie Zhou, Jiwen Lu

llms agents nlp multimodal benchmarks

arXiv AI

Aug 11

UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents

arXiv:2607. 04425v2 Announce Type: replace-cross Abstract: Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction.

By Niu Lian, Tongbo Chen, Zhehao Yu, Chengzhen Duan, Fazhan Liu, Hui Liu, Pei Fu, Jian Luan, Heng Qu, Shu-Tao Xia, Jinpeng Wang

agents efficiency multimodal

arXiv AI

Aug 25

GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis

arXiv:2608.22847v1 Announce Type: new Abstract: Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottl...

By Long Zhang, Yuhan Chen, Chaoran Zhang, Wanxia Cao, Kun Huang, Pengzhi Gao, Wei Liu, Jian Luan, Chenliang Li, Lixin Zou

llms agents reinforcement-learning multimodal benchmarks

Hugging Face Trending Papers

Aug 24

GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis

Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottlenecked by two fundamental issues: current data s...

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Sep 2

GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

GUI-CC is a benchmark designed to assess the contextual consistency of GUI world models when used as agent environments, rather than just one‑step next‑screen predictors. It includes two tracks: an offline reference‑action track that rolls models along real mobile GUI trajectories, and an online agent‑loop track where fixed probing agents interact with model‑generated UIs. The benchmark evaluates transition fidelity, plausibility, contextual consistency, and task progress across 500 offline trajectory tasks and 200 online tasks spanning 30 mobile apps.

By Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong

agents benchmarks

Hugging Face Trending Papers

Aug 6

StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions.

llms agents fine-tuning efficiency multimodal

arXiv AI

Aug 7

StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

arXiv:2608. 05587v1 Announce Type: new Abstract: Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution.

By Linqiang Guo (Peter), Wei Liu (Peter), Li Gu (Peter), Yang Wang (Peter), Tse-Hsun (Peter), Chen

llms agents fine-tuning efficiency multimodal

arXiv Computer Vision

Sep 21

MintAct: A Unified Visual Agent for Digital Environments

arXiv:2609.22083v1 Announce Type: new Abstract: We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual too...

By Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege \"Ozsoy, Kaixin Ma, Vishwesh Kirthivasan, O\u{g}uzhan Fatih Kar, Roman Bachmann, Anders Boesen Lindbo Larsen, Afshin Dehghan

llms agents reinforcement-learning multimodal benchmarks

Hugging Face Trending Papers

2d ago

AutoGUIWorld: Image Generators as Visual World Models for GUI Agent

GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows. However, the diversity of available tr...

agents fine-tuning

arXiv AI

Jul 1

Xiaomi-GUI-0 Technical Report

arXiv:2606. 31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation.

By Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao, Niu Lian, Fazhan Liu, Hui Liu, Heng Qu, Qinzhuo Wu, Zhehao Yu, Tongbo Chen, Shiqi Cui, Anan Du, Shukai Jia, Yuanfa Li, Yike Liu, Wenchao Lu, Haoyuan Sun, Jiatong Sun, Cheng Tan, Yajie Wang, Changqiao Wu, Tao Xiong, Jiahui Yang, Yuxuan Yuan, Ruoceng Zhang, Shaojie Zhang, Jian Zhu, Jian Luan, Cong Zou

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jun 30

GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

arXiv:2606. 29705v1 Announce Type: new Abstract: Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models.

By Sunqi Fan, Lingshan Chen, Runqi Yin, Qingle Liu, Yongming Rao, Meng-Hao Guo, Shi-Min Hu

agents reinforcement-learning

Related stories