Reinforcement learning

Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.

6,856 stories · RSS feed

arXiv Machine Learning
Sep 21

Survival Reinforcement Learning: Toward Scalable Self-Supervised RL

The paper introduces Survival Reinforcement Learning (SRL), an online classification-based method that extends the survival value learning framework to maximize an agent’s dwell time at target goals. SRL addresses limitations of contrastive reinforcement learning (CRL) in long-horizon, goal-conditioned tasks by avoiding the uniformity-tolerance dilemma and reducing undesirable bang-bang control behaviors. Across robotic benchmarks, SRL matches CRL on manipulation tasks and outperforms it by 2x to 8x on stable, long-horizon locomotion tasks, suggesting classification-based approaches are a promising direction for scaling reinforcement learning.

By Franki Nguimatsia-Tiofack, Fabian Schramm, Th\'eotime Le Hellard, Justin Carpentier
arXiv AI
Sep 21

CodeMidas: Scaling Agentic Coding RL Environments from Code Itself

arXiv:2609.22068v1 Announce Type: new Abstract: Training capable coding agents via reinforcement learning (RL) requires diverse tasks with reliable verifiers. Open-source codebases offer a rich sourc...

By Bowen Ye, Lei Li, Shicheng Li, Zihao Yue, Linghao Zhang, Hanglong Lv, Yuanxin Liu, Wenhan Ma, Hao Tian, Rang Li, Jinhao Dong, Yikai Zhao, Xiangwei Deng, Hailin Zhang, Liang Zhao, Qi Liu, Lingpeng Kong, Tong Yang, Fuli Luo
arXiv AI
Sep 21

Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generation

The paper introduces the COmmunity-centered Peer Engaged Support (COPES) dataset and a three‑axis evaluation framework to gauge how well Large Language Models (LLMs) align with community perspectives on mental‑health support queries. Experiments show that fine‑tuning LLMs on COPES improves strategy alignment and emotion‑tone alignment by over 50% for general‑purpose models, yet these gains are uneven across subreddits and coping strategies. The study also finds that post‑training shifts the model’s recommendations toward problem‑focused advice while reducing emotion‑focused responses, indicating persistent disparities in performance across different communities and needs.

By Mohit Chandra, Nabin Kim, Eli Min, Aamogh Sawant, Tanmay Sutar, Munmun De Choudhury
arXiv AI
Sep 21

Deep Reinforcement Learning with Buffered Quantile Objectives

The paper introduces Deep-BQRL, a model‑free distributional reinforcement‑learning framework that extends buffered‑quantile learning to neural function approximation. It learns conditional return quantiles from sampled transitions, constructs buffered action scores, and uses ensemble disagreement for exploration, enabling risk‑sensitive decision‑making without explicit return‑law planning. Experiments on asset‑selling and slippery FrozenLake show that Deep‑BQRL achieves smaller mean cumulative point‑quantile policy gaps than PPO and TRPO, while illustrating interpretable risk‑sensitive stopping decisions.

By Mohammad Alipour-vaezi, Sajad Khodadadian
arXiv AI
Sep 21

2nd Place Solution to the HANDS 2026 Workshop Challenge-Dexterous Grasp Motion Track: Single-Shot Trajectory Warping for Grasp Motion Generation

The paper reports a 2nd place solution for the HANDS 2026 Dexterous Grasp Motion track, targeting the 12‑DoF LinkerHand O6. The method warps a single successful GraspM3 demonstration into a 12‑D trajectory, avoiding step‑by‑step policy generation, and is trained with one‑step PPO across 4,824 objects. It achieved 94.61% success on the easy track and 57.18% on the hard track of the private test set.

By Muneeb A. Khan, Woojin Kim, Shinwoo Kim, Muhammad Munsif, Binod Bhattarai, Seungryul Baek
arXiv AI
Sep 21

OneBid: A Unified Auto-Bidding Foundation Model for Diverse oCPX Advertising Scenarios

OneBid is a unified auto‑bidding foundation model that consolidates diverse cost‑per‑X (oCPX) advertising scenarios into a single framework. It builds on Decision Transformer by conditioning on two atomic signals—Return‑to‑Go for conversion value and Cost‑to‑Go for cost ratio—and incorporates value‑aware regularization. A sequence‑level Mixture‑of‑Experts architecture captures cross‑scenario knowledge while preserving low latency, and a Critic‑guided Relative Offline Policy optimization (CROP) aligns the backbone with scenario‑specific preferences without unsafe online exploration. In production at Kuaishou, OneBid achieved a 2.2% overall ADVV increase and up to 13.1% in the ROAS scenario.

By Yewen Li, Peng Jiang, Yitian Li, Pengfei Lv, Xialong Liu, Peng Jiang, Qingpeng Cai
arXiv AI
Sep 21

Potential-Field Action Representation for Reinforcement Learning in Contact-Rich Manipulation

The paper introduces PA‑RL, a reinforcement‑learning framework that uses artificial potential fields as the action representation for contact‑rich robotic manipulation. Instead of directly commanding motion, the policy adjusts potential‑field parameters, which a Cartesian impedance controller then executes, decoupling task strategy from low‑level control. In peg‑in‑hole experiments, PA‑RL achieved a 100% success rate in simulation, outperformed baselines in torque and acceleration variation, and transferred to a real robot without fine‑tuning.

By Xinyu Liu, G\"okhan Solak, Arash Ajoudani
arXiv AI
Sep 21

SynthDemo-RL: Breaking the Zero-Reward Barrier in VLA Adaptation with LLM-Guided Synthetic Demonstrations

SynthDemo‑RL introduces a teacher‑student framework that uses an automated teacher to generate successful manipulation trajectories from simulator‑privileged state, which are then distilled into a Vision‑Language‑Action (VLA) student via supervised fine‑tuning. The student is further refined with PPO using binary task‑success rewards. On the LIBERO‑PRO benchmark, SynthDemo‑RL rescues all 27 previously unsolvable tasks and achieves near‑perfect success rates, matching performance that would otherwise require human demonstrations.

By Hiroaki Kingetsu, Hiroaki Kurihara, Kaoru Yokoo, Kenji Fukumizu, Manohar Kaul
arXiv AI
Sep 21

BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models

BoostAPR is a three-stage framework that improves automated program repair by using execution-grounded reinforcement learning with dual reward models. The approach first fine‑tunes a model on execution‑verified demonstrations, then trains a sequence‑level assessor and a line‑level credit allocator from execution outcomes, and finally applies PPO optimization where the line‑level model redistributes rewards to critical edit regions. Evaluated on SWE‑Gym and four benchmarks, BoostAPR achieves significant gains, including 40.7% on SWE‑bench Verified and 95.0% on QuixBugs, demonstrating strong cross‑language generalization.

By Yuanhao Li, Hongbo Wang, Xiaotang Shang, Xunzhu Tang, Yiming Cao, Xuhong Chen
arXiv AI
Sep 21

Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning

The paper introduces CoDRA, a cost-to-disturbance ratio approach for adversarial reinforcement learning that balances controller performance and disturbance exposure without extra penalty terms. CoDRA uses a self‑normalized actor–critic update, scaling value terms by a stop‑gradient normalization constant derived from the current batch. Experiments on MuJoCo pendulum tasks show that CoDRA achieves the lowest cost across a range of forces and masses, outperforming other methods especially on the more challenging InvertedDoublePendulum environment.

By Taeho Lee, Donghwan Lee
arXiv Machine Learning
Sep 21

Efficient Bayes-Adaptive Reinforcement Learning with Temporal Logic Specifications

The paper introduces an end‑to‑end model‑based reinforcement learning algorithm that synthesises policies satisfying Linear Temporal Logic (LTL) specifications in unknown environments. It synchronises a Limit‑Deterministic Büchi Automaton (LDBA) with a Bayes‑Adaptive Markov Decision Process (BAMDP) and proposes a novel Bayes‑Adaptive Monte‑Carlo Planning (BAMCP) method for approximate Bayes‑optimal strategy synthesis. Experiments on finite and infinite‑horizon tasks show improved property satisfaction and sample efficiency compared to model‑free baselines, and ablation studies confirm the advantage of the new BAMCP over classical variants, including reduced task violations in cautious RL settings.

By Jonathan Hau, Alessandro Abate
arXiv AI
Sep 21

Reinforcement Learning under External Influence: Guarantees, Algorithms, and Sample Complexity

The paper investigates reinforcement learning in Markov decision processes whose dynamics are perturbed by non‑Markovian external events. It identifies conditions that make the problem tractable by limiting consideration to a finite history of events, and proposes a policy iteration algorithm that learns state‑dependent policies conditioned on this history. The authors provide theoretical guarantees for policy improvement, analyze sample complexity for least‑squares evaluation and improvement, and extend their results to discrete‑time Hawkes processes with Gaussian marks, validating their approach with experiments in control environments.

By Ranga Shaarad Ayyagari, Revanth Raj Eega, Ambedkar Dukkipati
arXiv AI
Sep 21

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

OmniVChat defines a native audio‑visual dialogue task where models receive raw audio and video from a user and produce text responses, eliminating the need for separate text queries or speech recognition. To address data scarcity and evaluation challenges, the authors introduce OmniVChat‑Studio, a multi‑agent engine that synthesizes single‑ and multi‑turn dialogues, and OmniVChat‑Bench, a benchmark assessing models across five dialogue ability categories. They also propose OmniVChat‑RL, a reinforcement‑learning reward that balances reply correctness, efficiency, and style, and demonstrate that training Qwen3‑Omni‑Instruct with this reward on synthesized data improves performance on both synthetic and human‑recorded benchmarks.

By Haolin He, Yunfei Chu, Qi Chen, Wen Huang, Yuan Feng, Muzhi Zhu, Zheqi Dai, Haoning Xu, Dongchao Yang, Chunyat Wu, Zining Liang, Zhengxi Liu, Xiquan Li, Xie Chen, Xize Cheng, Qize Yang, Jin Xu, Qiuqiang Kong
arXiv Machine Learning
Sep 21

ASGARD: Action-Space Guard for UAV Resilience via Reinforcement Learning

ASGARD is a two‑phase teacher‑student framework that protects reinforcement‑learning controllers for UAVs from action‑space attacks. In the teacher phase, an encoder fuses the UAV’s physical state with privileged attack information to generate an action‑attack‑aware latent representation, which trains both the control policy and a monitor that corrects actions before they reach the actuators. The student phase then learns to replicate the encoder and monitor using only the UAV’s physical state history, enabling on‑board resilience. Experiments show that ASGARD remains effective against various attack scenarios, including unseen and stealthy attacks, allowing UAV missions to complete successfully.

By Mohsen Salehi, Karthik Pattabiraman