Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2609.14739v1 Announce Type: cross Abstract: Large language models are increasingly used in high-stakes domains such as law, where systems must ground their reasoning in retrieved evidence and a...
The paper introduces Drift-Constrained Optimization (DCO), a framework that treats behavioral drift during fine‑tuning of instruction models as a bounded constraint rather than an uncontrolled side effect. By defining a drift budget, the authors reformulate fine‑tuning as a direction‑selection problem, showing that choosing different update directions can qualitatively change outcomes. Experiments on Qwen3 models demonstrate that carefully selected directions improve scientific reasoning and multilingual translation while preserving reasoning capabilities and general performance.
arXiv:2609.15539v1 Announce Type: cross Abstract: Finding policies for Markov Decision Processes (MDPs) is a central problem in areas such as Reinforcement Learning and Operations Research. Here, we...
arXiv:2609.14998v1 Announce Type: new Abstract: Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptab...
arXiv:2609.15883v1 Announce Type: cross Abstract: Offline reinforcement learning aims to learn a policy solely from fixed datasets, which often contain multimodal action distributions. Flow policies...
arXiv:2609.15066v1 Announce Type: cross Abstract: We present Salesforce Koa, an enterprise language model built by post-training the open-weight Nemotron-3-Super-120B foundation model with reinforcem...
Multimodal large language models (MLLMs) have made strong progress on visual question answering and image captioning, yet they still produce fluent claims about objects, attributes, or relations that...
Finding policies for Markov Decision Processes (MDPs) is a central problem in areas such as Reinforcement Learning and Operations Research. Here, we have to repeatedly choose an action that should be...
arXiv:2609.13058v1 Announce Type: new Abstract: Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models hav...
The paper presents a study on adaptive chemotherapy control using deep reinforcement learning (DRL) to address tumor heterogeneity and drug resistance. Closed‑loop DRL dosing policies—continuous (TD3) and discrete (DQN)—are trained on a high‑dimensional heterogeneous tumor model and benchmarked against a Pontryagin's Maximum Principle (PMP) open‑loop solution. Across a 100‑patient virtual cohort with ±10% parameter perturbations, TD3 achieves higher average tumor reduction, while DQN offers tighter inter‑patient dosing consistency, highlighting an efficacy‑consistency trade‑off. The work assumes full observation of tumor subpopulations, noting that clinical translation will require handling sparse, noisy measurements.
The paper introduces GACA, a critic‑free reinforcement learning estimator that adapts credit assignment granularity based on a step‑level uncertainty proxy. GACA assigns higher weight to fine‑grained signals for steps with above‑average negative log‑likelihood, while relying on episode‑level signals for less uncertain steps, improving task success on ALFWorld and WebShop for 1.5B and 7B language models. The authors provide a risk decomposition, a conditional bound on action‑value variation, and an error‑projection analysis to justify the method’s effectiveness.
arXiv:2601.20433v4 Announce Type: replace Abstract: Deepfake detection is a widely researched topic that is crucial for combating the spread of malicious content, with existing methods mainly modelin...
arXiv:2609.12230v1 Announce Type: new Abstract: Question-answering often requires reasoning across multiple connected facts rather than retrieving a single isolated relation. Knowledge graphs (KGs) p...
The paper introduces SELECT, an algorithmic framework for satisficing regret minimization in bandit problems, achieving constant expected satisficing regret when a satisficing arm exists. A variant, SELECT‑LITE, further ensures a light‑tailed satisficing regret distribution while maintaining constant expected regret in the realizable case and sub‑linear standard regret otherwise. Experiments on synthetic data and a real‑world dynamic pricing scenario demonstrate the practical effectiveness of both algorithms.
The paper introduces CLAW, a method that uses a hypernetwork to generate low‑rank adapters for world models during test time, enabling efficient adaptation to new environments with only a few episodes of interaction. By jointly pretraining the hypernetwork and base model on simulated adaptations, CLAW balances computational efficiency and expressivity, outperforming both in‑context learning and gradient‑based adaptation in locomotion and manipulation tasks. The approach also mitigates overfitting in data‑scarce regimes and demonstrates that the benefit stems from expressive adapters rather than context conditioning.
The paper introduces an adversarial reinforcement learning framework that learns the sparsest Denial-of-Service (DoS) attack schedule capable of destabilizing self‑triggered reinforcement learning controllers (RL‑STC). It proves a lower bound on the minimum number of jamming actions needed to force a crash and demonstrates that the learned adversary consistently defeats four different defenders—one LQR and three RL‑STC—across Pendulum, CartPole, and Quadrotor2D environments, outperforming greedy and periodic baselines in jam‑time‑per‑failure. The study also shows that the adversary remains effective under Gaussian observation noise and limited state information.
The paper introduces MTD3-BC, a model‑free offline reinforcement learning algorithm that optimizes yaw control for wind farms amid changing wind directions. By learning from a pre‑collected dataset and incorporating an action consistency term, it reduces the need for extensive simulator interactions. Experimental wind‑tunnel tests show that MTD3‑BC improves farm‑level power output by about 10% compared to a greedy baseline and matches a model‑based benchmark, all while cutting training costs dramatically.
SaFeR-Steer is a progressive multi‑turn alignment framework that uses staged synthetic bootstrapping and tutor‑in‑the‑loop GRPO to train a single student model against adaptive, on‑policy attacks. It introduces Trajectory‑Consistent Summative Reward (TCSR) to ensure that low‑quality turns impact the overall trajectory reward. The authors release the STEER dataset and demonstrate that SaFeR‑Steer significantly improves safety and helpfulness on both single‑turn and multi‑turn benchmarks for Qwen2.5‑VL models.
The paper introduces Decision-Flow Sampling (DF‑Sample), a training‑free, data‑free inference framework that builds a hierarchical reasoning tree, evaluates entire trajectories, and back‑propagates utilities to guide branching decisions. Unlike local step‑wise sampling, DF‑Sample explicitly assesses global paths, enabling it to recover high‑quality, low‑probability reasoning chains that standard decoding misses. On the GPQA benchmark, DF‑Sample attains 45.6% accuracy, outperforming power sampling (38.9%) and GRPO (39.9%) and consistently surpassing baselines across multiple models and benchmarks, demonstrating significant latent reasoning potential in pretrained LLMs.
The paper proposes a reinforcement learning (RL) fine‑tuning framework for electronic health record (EHR) foundation models, treating them as generative policies over patient trajectories. By framing clinical prediction tasks such as hospital readmission as event‑conditioned, time‑windowed reasoning problems and designing time‑aware, rollout‑sensitive rewards, the authors show that RL fine‑tuning consistently outperforms pre‑trained backbones and strong baselines. The approach enables smaller models to surpass larger pre‑trained models in data‑limited settings, induces positive transfer across tasks, and produces trajectories with stronger structural and semantic alignment to ground truth, improving downstream utility.