arXiv AI
Jul 28
Scaling GUI Agents with Visual State Transitions
arXiv:2607. 24112v1 Announce Type: new Abstract: We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents.
Scaling GUI Agents with Visual State Transitions
We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions).
Related stories
arXiv AI
Jul 21
EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
arXiv:2607. 17050v1 Announce Type: cross Abstract: GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery.
arXiv AI
Aug 11
UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents
arXiv:2607. 04425v2 Announce Type: replace-cross Abstract: Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction.
Hugging Face Trending Papers
Aug 6
StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions.
arXiv AI
Aug 7
StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
arXiv:2608. 05587v1 Announce Type: new Abstract: Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution.
arXiv AI
Jul 1
Xiaomi-GUI-0 Technical Report
arXiv:2606. 31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation.
arXiv AI
Jun 30
GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots
arXiv:2606. 29705v1 Announce Type: new Abstract: Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models.
arXiv AI
Jul 2
KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learning
arXiv:2601. 14232v2 Announce Type: replace-cross Abstract: Pixel-based reinforcement learning agents often fail under purely visual distribution shift even when latent dynamics and rewards are unchanged, but existing benchmarks entangle multiple sources of shift and hinder systematic analysis.
Hugging Face Trending Papers
Jul 20
SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis
Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks. However, progress is limited by the lack of high-coverage, long-horizon interaction trajectories collected from element-rich and rapidly evolving apps.
arXiv Machine Learning
Jun 5
Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning
arXiv:2512. 09706v2 Announce Type: replace Abstract: The paradigm of agentic AI is shifting from engineered complex workflows to post-training native models.
arXiv AI
Jun 15
GUITrans2Act: Understanding User Operational Behaviors from Mobile GUI Interactions with Vision-Language Models
arXiv:2606. 12817v2 Announce Type: replace Abstract: Understanding the digital world on mobile devices is shifting from static UI perception to dynamic action comprehension.
arXiv AI
Jul 7
UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning
arXiv:2607. 04425v1 Announce Type: cross Abstract: Recent advances in multimodal foundation models and agent systems have driven GUI agents from single-platform task execution toward cross-platform interaction.