Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
The article discusses the challenge of teaching foundation models new skills without disrupting existing capabilities. It proposes a reinforcement learning-based method called on‑policy self‑distillation as a potential solution. The piece explores how this approach could transform post‑training processes for large language models.
arXiv:2610.28785v1 Announce Type: new Abstract: Training tool-using language models requires costly interaction, yet identical rewards within a rollout group provide no group-relative learning signal...
arXiv:2610.29411v1 Announce Type: new Abstract: Outcome-predictive signals are often treated as promising optimization targets in reinforcement learning, but predictive validity and optimization util...
arXiv:2610.29427v1 Announce Type: new Abstract: Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance of LLM agents on complex, long-horizon interactiv...
arXiv:2610.29380v1 Announce Type: new Abstract: Practical image generation requires both human preference alignment and few-step efficiency, yet combining reinforcement learning (RL) with distillatio...
Flat-RLVR introduces a sharpness-aware approach to reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs). By targeting flat reward regions, the method reduces distribution sharpening, thereby preserving sample diversity while maintaining high single-attempt accuracy. The technique combines flat reward optimization with group-relative policy optimization (GRPO) variants to improve the trade-off between sampling efficiency and reasoning capacity.
DISRL is a Dual-Scale Incremental Semi-Supervised Reinforcement Learning framework designed for dynamic local community detection. It tackles temporal supervision uncertainty by generating seed-specific guidance from sparse known communities and mitigates unbounded error cascades through a dual-scale community exploration module that separates short-term adaptation from long-term parameter consolidation. Experiments on a newly constructed dynamic graph benchmark show DISRL consistently outperforms state‑of‑the‑art baselines across various dynamic settings.
The paper introduces DriveMLP, a lightweight MLP policy for autonomous driving that is trained using an Expert‑Guided RL (EGRL) framework. EGRL leverages reinforcement learning solely as an optimization tool, guided by a trajectory‑distance reward derived from logged expert data, and the study shows that the training strategy itself yields significant performance gains over model complexity. DriveMLP achieves an 86.51 score on the nuPlan Val14 Reactive benchmark, surpassing the previous best IL planner by 3.2 points while offering more than 15× faster inference.
CORD is a conflict‑aware optimization framework that addresses gradient cancellation between task rewards and teacher guidance in joint reinforcement learning and self‑distillation. It separates the protected objective from the projection trigger, applies a nearest‑point closed‑form update per mini‑batch, and offers gradient diagnostics to identify and correct conflicts. Experiments across three training recipes show that CORD improves ALFWorld success by 1.5–15.9 percentage points and raises WebShop SDAR success from 60.7% to 64.6%.
The paper studies the trade‑off between maximizing rewards and accurately estimating current treatment effects in nonstationary multi‑armed bandits. For $K$‑armed bandits with H"older‑smooth mean rewards, the authors derive minimax bounds on expected dynamic regret under a high‑probability simultaneous coverage requirement for all current effects. They propose the TRACE algorithm, which achieves a regret–precision trade‑off that separates reward‑learning regret from the unavoidable cost of refreshing current‑effect estimates, and extend the method with an empirical‑Bernstein e‑process to adapt exploration while preserving worst‑case guarantees.
The paper introduces an energy-based transfer learning approach for reinforcement learning, aiming to improve sample efficiency in continual learning scenarios. By using out-of-distribution detection, the method selectively provides teacher guidance only in states that fall within the teacher’s training distribution, preventing sub-optimal bias when tasks differ significantly. The authors theoretically connect energy scores to state-visitation patterns under on-policy RL and demonstrate empirical gains in both single-task and multi-task settings.
ChemBART is a BART‑based language model trained on holistic reaction expressions (Reactant > Reagent > Product) using a mask‑filling objective. It can solve diverse chemical tasks such as precursor/reagent generation, temperature‑yield regression, and property classification, and it is integrated with Monte Carlo tree search to design multi‑step synthesis routes. The model’s predictions were experimentally validated, yielding shorter synthetic pathways with over 20 % higher yield than literature benchmarks.
The paper demonstrates that a minimal foraging swarm can approach the explore/exploit optimum of reinforcement learning without any learner or reward signal. Agents use heavy‑tailed Levy flights to explore, recruit neighbors via a threshold cascade on their proximity network, and disperse as patch crowds, creating a cycle that balances exploration and exploitation. The swarm attains the shape of the ideal‑free distribution (committed foragers track richness with slope 0.94) and reaches this allocation dynamically in about 150 steps, though it remains slightly exploitable and does not achieve the exact equilibrium.
LIFT introduces Lift-Bench, a new instruction‑following benchmark that asks language models to draft a document based on a topic and a structured outline, with constraints scoped to specific sections for finer control. The benchmark includes both verifiable and judged constraints and a training counterpart, Lift‑Train, designed to reduce reward hacking by providing both main‑task and instruction‑following constraints. Experiments show that state‑of‑the‑art models struggle with the harder subsets, and training on Lift‑Train is less prone to reward hacking than traditional IF training.
The paper introduces BiasGuard‑D, a system that uses two adversarial language‑model agents from different families to exchange typed claim documents, with a third model quorum voting to adjudicate. A combined referential‑plus‑semantic invariant validates the verdict, and the iterative deliberation layer shifts remediation from stereotypes to hallucinations, improving right‑answer recovery by 49 percentage points on HaluEval QA. All claims, votes, and instructions are stored as typed records, providing an auditable provenance trail for operators under emerging AI regulations.
The study investigates how post‑training alignment techniques affect small language models (SLMs) for generating medication leaflets from structured biomedical data. Four alignment methods—supervised fine‑tuning (SFT), direct preference optimization (DPO), odds ratio preference optimization (ORPO), and group relative policy optimization (GRPO)—are evaluated using Qwen‑2.5 models, with a new FDALeaflets dataset derived from OpenFDA drug labels for zero‑shot cross‑regulatory testing. Results show that aligned SLMs can outperform GPT‑5 on reference‑based metrics, while ORPO and GRPO achieve the highest semantic textual similarity scores in‑domain and on FDALeaflets, respectively, highlighting the importance of alignment for semantic fidelity and generalization.
The paper introduces Dynalize, an online penalty system that mitigates reward hacking in preference reward models during policy training. By having a language model periodically analyze rollouts and generate penalties for behaviors the reward model inadvertently rewards, Dynalize prevents the policy from inflating scores without improving quality. Experiments in clinical counseling and creative writing show that Dynalize sustains longer training and yields better downstream performance, suppressing undesirable patterns even in multi‑turn settings and after retraining the reward model.
The paper introduces a contextual bandit algorithm that incorporates concavity constraints into a Bayesian Gaussian Process posterior. By combining a concavity-preserving regression spline basis with a constrained GP, the authors create a tractable shape-constrained estimator and a UCB algorithm. The resulting method achieves regret bounds that are at least as good as standard GP-UCB and can be tighter when concavity information is informative, as shown by experiments on benchmarks and a Warfarin dosing application.
Doob-Based Generative Policy Optimization (DoobPO) introduces a likelihood‑free framework for fine‑tuning generative policies such as diffusion and flow models. It first learns a lightweight ratio model to represent the desired policy update and then distills this ratio‑tilted target policy into the generative policy via conditional drift matching. The method supports standard policy improvement rules like PPO and GRPO, proves gradient equivalence to an ideal Doob drift matching objective, and demonstrates strong performance on MuJoCo continuous‑control benchmarks.
BIRD‑RL is a new agentic reinforcement learning framework designed to scale training of small language models (SLMs) in data‑centric, stateful environments such as SQL debugging. It introduces a four‑stage curriculum that alternates between improving agent reasoning and trajectory‑adaptive exploration, and relies on a Trajectory‑scoped Persistent Execution Infrastructure to safely reuse database replicas while maintaining trajectory isolation. This approach yields a 12.8× improvement in spatial efficiency and enables 7B and 14B models to achieve competitive success rates on the BIRD‑CRITIC‑SQLite benchmark, matching proprietary agents like Claude‑Opus‑4.6.