arXiv AI

KAGE-Bench: Fast Known-Axis Visual Generalization Evaluation for Reinforcement Learning

arXiv:2601. 14232v2 Announce Type: replace-cross Abstract: Pixel-based reinforcement learning agents often fail under purely visual distribution shift even when latent dynamics and rewards are unchanged, but existing benchmarks entangle multiple sources of shift and hinder systematic analysis.

arXiv AI

Jul 28

Scaling GUI Agents with Visual State Transitions

arXiv:2607. 24112v1 Announce Type: new Abstract: We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents.

By Xiangyan Liu, Kaixin Li, Haonan Wang, Biao Wu, Meng Fang, Longxu Dou, Chao Du, Michael Qizhe Shieh, Tianyu Pang

agents fine-tuning multimodal benchmarks

arXiv AI

Jun 16

QPILOTS: Efficient Test-Time Q-Steering for Flow Policies

arXiv:2606. 14801v1 Announce Type: cross Abstract: Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult.

By Yifan Ruan, Chenyang Cao, Andreas Burger, Ali Pesaranghader, Kaveh Kamali, Jaehong Kim, Nandita Vijaykumar, Alan Aspuru-Guzik, Igor Gilitschenski, Nicholas Rhinehart

diffusion reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 7

Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control

arXiv:2608. 05989v1 Announce Type: new Abstract: Sample-efficient policy learning from pixels is a long-standing challenge in reinforcement learning (RL).

By Xinwei Liu, Junyuan Liang, Jianting Zhang, Wuhui Chen

diffusion reinforcement-learning benchmarks

Hugging Face Trending Papers

Jul 27

Scaling GUI Agents with Visual State Transitions

We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions).

agents fine-tuning multimodal benchmarks

Hugging Face Trending Papers

Aug 6

Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control

Sample-efficient policy learning from pixels is a long-standing challenge in reinforcement learning (RL). Recent dynamics-based representation learning methods have significantly improved the sample efficiency of model-free visual RL by learning dynamics-aware representations through auxiliary prediction performed either in latent space (self-prediction) or observation space (observation prediction).

diffusion reinforcement-learning benchmarks

arXiv AI

Jun 30

GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

arXiv:2606. 29705v1 Announce Type: new Abstract: Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models.

By Sunqi Fan, Lingshan Chen, Runqi Yin, Qingle Liu, Yongming Rao, Meng-Hao Guo, Shi-Min Hu

agents reinforcement-learning

arXiv AI

Jun 10

A History-Aware Visually Grounded Critic for Computer Use Agents

arXiv:2606. 11078v1 Announce Type: new Abstract: Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments.

By Jaewoo Lee, Zaid Khan, Archiki Prasad, Justin Chih-Yao Chen, Supriyo Chakraborty, Kartik Balasubramaniam, Sambit Sahu, Elias Stengel-Eskin, Hyunji Lee, Mohit Bansal

llms agents multimodal benchmarks

arXiv AI

Jul 21

EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding

arXiv:2607. 17050v1 Announce Type: cross Abstract: GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery.

By Yaohan Yang, Minglei Shi, Borui Zhang, Jie Zhou, Jiwen Lu

llms agents nlp multimodal benchmarks

arXiv AI

Jun 15

VISTA: View-Consistent Self-Verified Training for GUI Grounding

arXiv:2606. 14579v1 Announce Type: new Abstract: When applying Group Relative Policy Optimization (GRPO) for GUI Grounding, rollouts are sampled from a single screenshot view; groups often become either all failures on difficult instances or all successes on easy ones, yielding no useful relative advantage.

By Xinyu Qiu, Yunzhu Zhang, Heng Jia, Shuheng Shen, Changhua Meng, Linchao Zhu

reinforcement-learning benchmarks

arXiv AI

Jun 24

Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

arXiv:2606. 24515v1 Announce Type: new Abstract: Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces.

By Marta Sumyk, Oleksandr Kosovan

llms agents reinforcement-learning fine-tuning multimodal

Hugging Face Trending Papers

Jul 14

UniVR: Thinking in Visual Space for Unified Visual Reasoning

Learning broad world knowledge directly from raw visual data is a fundamental capability of intelligence. We introduce UniVR, the first investigation into simultaneously learning complex reasoning, fine-grained physical dynamics, and long-term planning from pure visual demonstrations.

reinforcement-learning robotics multimodal benchmarks

arXiv Machine Learning

Jul 31

S-GRPO: Unified Post-Training for Large Vision-Language Models

arXiv:2604. 16557v2 Announce Type: replace Abstract: Current post-training methodologies for adapting Large Vision-Language Models (LVLMs) generally fall into two paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).

By Yuming Yan, Kai Tang, Sihong Chen, Ke Xu, Dan Hu, Qun Yu, Pengfei Hu

llms reinforcement-learning fine-tuning multimodal

Related stories