arXiv AI

Training a Conditioned Video Game Agent on a VLM Annotated Dataset

arXiv:2608. 05954v1 Announce Type: new Abstract: Reinforcement Learning (RL) is a powerful but far from easy-to-use technique for policy learning.

arXiv AI

Aug 7

VLMs for Videogame Data Annotation

arXiv:2608. 05949v1 Announce Type: new Abstract: Vision Language Models (VLMs) and Artificial Intelligence (AI) agents have revolutionized how engineers approach complex problems in real-world applications.

By Katrin Schmid, Iuri Frosio

llms agents reinforcement-learning multimodal

arXiv AI

Jun 2

From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models

arXiv:2606. 00083v1 Announce Type: cross Abstract: Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics.

By Christian Gumbsch, Leonardo Barcellona, Lennard Sch\"unemann, Platon Karageorgis, Andrii Zadaianchuk, Zehao Wang, Sergey Zakharov, Fabien Despinoy, Rahaf Aljundi, Efstratios Gavves

llms reinforcement-learning robotics multimodal

arXiv AI

Jun 26

Automating Potential-based Reward Shaping with Vision Language Model Guidance

arXiv:2606. 27180v1 Announce Type: cross Abstract: Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory.

By Henrik M\"uller, Daniel Kudenko

llms agents reinforcement-learning multimodal

arXiv AI

Jul 10

Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs

arXiv:2607. 08193v1 Announce Type: cross Abstract: Open-ended curricula in Reinforcement Learning (RL) aim to train generally-capable agents by identifying tasks that facilitate learning increasingly complex skills.

By Lorenzo Pant\`e, Andrea Fanti, Roberto Capobianco

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Aug 5

Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning

arXiv:2608. 03875v1 Announce Type: cross Abstract: Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL).

By Pyrros Koussios, Chenhao Li, Xin Chen, Andreas Krause

llms diffusion reinforcement-learning fine-tuning multimodal safety

arXiv AI

Jun 24

Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

arXiv:2606. 24515v1 Announce Type: new Abstract: Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces.

By Marta Sumyk, Oleksandr Kosovan

llms agents reinforcement-learning fine-tuning multimodal

arXiv AI

Jul 24

TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

arXiv:2602. 19313v2 Announce Type: replace-cross Abstract: General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior.

By Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, Ranjay Krishna

llms robotics multimodal benchmarks

arXiv AI

Jun 4

Belief-Aware VLM Model for Human-like Reasoning

arXiv:2604. 09686v2 Announce Type: replace Abstract: Traditional neural network models for intent inference rely heavily on observable states and struggle to generalize across diverse tasks and dynamic environments.

By Anshul Nayak, Shahil Shaik, Yue Wang

llms diffusion reinforcement-learning multimodal

arXiv AI

Jun 30

Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models

arXiv:2606. 29892v1 Announce Type: cross Abstract: Reinforcement learning (RL) has become indispensable for pushing Vision-Language-Action Models (VLAs) beyond static imitation learning.

By Siyao Chen, Jiakang Yuan, Jiaxin Wang, Tao Chen

reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jul 8

Supervised Reward Inference

arXiv:2502. 18447v2 Announce Type: replace Abstract: Existing approaches to reward inference typically assume that humans provide demonstrations according to specific behavior models.

By Will Schwarzer, Jordan Schneider, Philip S. Thomas, Scott Niekum

robotics benchmarks

arXiv Machine Learning

Jun 2

RDA: Reward Design Agent for Reinforcement Learning

arXiv:2606. 01672v1 Announce Type: new Abstract: Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions.

By Hojoon Lee, Ajay Subramanian, Ben Abbatematteo, Vijay Veerabadran, Pedro Matias, Karl Ridgeway, Nitin Kamra

llms agents reinforcement-learning robotics multimodal

arXiv Machine Learning

Jun 2

Towards Sparse Video Understanding and Reasoning

arXiv:2602. 13602v2 Announce Type: replace-cross Abstract: We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA).

By Chenwei Xu, Zhen Ye, Shang Wu, Weijian Li, Zihan Wang, Zhuofan Xia, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han Liu

llms agents nlp fine-tuning multimodal benchmarks

Related stories