← Back to all news

Reinforcement learning

Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.

6,856 stories · RSS feed

arXiv Machine Learning
Sep 22

Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

Swiss-Knife is a framework that extends decode‑time alignment for frozen language models by treating the alignment specification as a runtime object. It introduces hot‑swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule, and characterises admissible aggregation operators with a representation theorem. In experiments, Swiss‑Knife paired with DPO‑LoRA blades and an uncertainty‑aware pairwise tournament outperforms six existing decode‑time methods, achieving a higher harmonic F1 score, lower refusal rate, and faster objective reconfiguration.

By Agnibh Karmakar, Mayur Parvatikar, Shreyash Dhoot, Amit Dhanda, Aman Chadha, Kapil Wanaskar, Vinija Jain, Amitava Das
llmsreinforcement-learningfine-tuningsafety
More like this →
arXiv Computation and Language
Sep 22

Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting

The paper investigates time‑incremental continued pretraining (CPT) of large language models using web‑scale data that overlaps across snapshots. Across six open‑weight models, CPT improves factual recall without catastrophic forgetting, while the cost is negligible and data quality outweighs quantity. The study identifies optimal learning rates, shows LoRA can match full CPT, and demonstrates that CPT gains transfer to fine‑tuned models.

By F{\i}rat \"Oncel, Salman Hussain Ali, Mirco Ravanelli, Cem Subakan, \c{C}a\u{g}atay Y{\i}ld{\i}z
llmsreinforcement-learningfine-tuningdata-engineering
More like this →
arXiv Machine Learning
Sep 22

Computationally efficient safe exploration in reinforcement learning

The paper introduces “CoLSafe-MDP”, a reinforcement learning algorithm that ensures safe exploration in constrained Markov decision processes. It replaces computationally heavy Gaussian process methods with a Nadaraya-Watson estimator, achieving constant-time scaling for estimate bounds. The authors evaluate the algorithm on a grid-based environment and on observational Martian terrain data.

By Shreeram Murali, Shankar A. Deka, Dominik Baumann
reinforcement-learningsafety
More like this →
arXiv Machine Learning
Sep 22

Robust Market Making with Hawkes Order Flow and Price Impact via Adversarial Reinforcement Learning

The paper proposes an adversarial reinforcement‑learning framework for market making that incorporates Hawkes‑process driven order arrivals and trade‑induced price impact, addressing limitations of prior Poisson‑based models. An LSTM module captures temporal dependencies in recent observations to handle increased non‑stationarity, and the authors analyze equilibrium properties and introduce a robustness evaluation protocol focused on the left tail of returns. Experiments across diverse market regimes demonstrate that the method improves left‑tail performance, especially under strong Hawkes excitation and moderate price impact, without relying on a terminal inventory bias.

By Hao Yang, Zhenguo Xu
reinforcement-learningsafety
More like this →
arXiv Machine Learning
Sep 22

Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization

Prioritized Rollouts for Efficient World Model-based Vision-Language-Action Policy Optimization introduces U‑GROW, a lightweight sampling layer that directs more model rollouts toward states with high policy uncertainty, identified as decision‑sensitive stages where small action differences can alter task outcomes. By modifying only the branched‑start distribution, U‑GROW can be integrated into existing model‑based reinforcement learning pipelines without changing the policy optimization objective. Experiments on simulated and real‑world manipulation tasks demonstrate that U‑GROW improves the efficiency and effectiveness of policy optimization for Vision‑Language‑Action models.

By Yifei Sheng, Haoxiang Ren, Zhilong Zhang, Haonan Wang, Runjie Xu, Yihao Sun, Nan Tang, Zhichao Wu, Lei Yuan, Haoxin Lin, Yang Yu
reinforcement-learningroboticsfine-tuningmultimodal
More like this →
arXiv Machine Learning
Sep 22

Proximal Residual Value Functions for Consistent Planning and Real-Time Execution

The paper introduces proximal residual value functions for two‑timescale decision systems, where a planning layer supplies a continuation‑value function to a real‑time optimizer that allocates resources, with inventory placement as a motivating example. The authors propose an end‑to‑end reinforcement learning method that learns a convex residual added to a strictly convex potential, enabling well‑posed optimization and end‑to‑end differentiation while maintaining an explicit convex objective for real‑time execution. They also provide necessary and sufficient conditions for smooth value functions to produce decisions consistent across planning and execution timescales, and demonstrate a 5.0% reduction in routing and transfer cost in an offline simulation using data from a large e‑commerce retailer.

By Harrison Waldon, Carson Eisenach, Akhil Bagaria, Daniel Russo, Dominique Perrault-Joncas, Alisha Zachariah, Dean Foster
reinforcement-learning
More like this →