The paper introduces SEE, a two-stage framework for generating long-horizon GUI agent trajectories. First, an exploration stage builds an explicit UI transition graph over screens and elements. Second, a graph-based synthesis stage composes diverse multi-step trajectories through planning and controlled sampling, preventing spurious cycles and enabling long-horizon composition. Across real-world apps, SEE produces trajectories averaging 14.8 steps and improves agent task success and generalization to unseen screens.
By Zhuohang Fan, Beichen Zhang, Yuanfa Li, Changqiao Wu, Wei Liu, Jian Luan, Weigang Zhang
arXiv:2607. 18046v1 Announce Type: new Abstract: Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks.
By Zhuohang Fan, Beichen Zhang, Yuanfa Li, Changqiao Wu, Wei Liu, Jian Luan, Weigang Zhang
GUI-CC is a benchmark designed to assess the contextual consistency of GUI world models when used as agent environments, rather than just one‑step next‑screen predictors. It includes two tracks: an offline reference‑action track that rolls models along real mobile GUI trajectories, and an online agent‑loop track where fixed probing agents interact with model‑generated UIs. The benchmark evaluates transition fidelity, plausibility, contextual consistency, and task progress across 500 offline trajectory tasks and 200 online tasks spanning 30 mobile apps.
By Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong
arXiv:2606. 12817v1 Announce Type: new Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.
By Yudong Zhang (Honor Device Co., Ltd), Lei Hu (Honor Device Co., Ltd), Daoyang Liu (The Chinese University of Hong Kong, Hong Kong, China), Jiawei Liu (Honor Device Co., Ltd), Yangfan Luo (Honor Device Co., Ltd), Xingyu Liu (Honor Device Co., Ltd), Zuojian Wang (Honor Device Co., Ltd), Zhilin Gao (Honor Device Co., Ltd)
GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows. However, the diversity of available tr...
arXiv:2606. 12817v2 Announce Type: replace Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.
By Yudong Zhang (Honor Device Co., Ltd), Lei Hu (Honor Device Co., Ltd), Daoyang Liu (The Chinese University of Hong Kong, Hong Kong, China), Jiawei Liu (Honor Device Co., Ltd), Yangfan Luo (Honor Device Co., Ltd), Zhilin Gao (Honor Device Co., Ltd), Zuojian Wang (Honor Device Co., Ltd)