Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2610.02835v1 Announce Type: new Abstract: During post-training of large language models (LLMs) with Reinforcement Learning with Verifiable Rewards (RLVR), GRPO-style algorithms can exhibit seve...
arXiv:2610.02700v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) supplies dense token-level feedback on trajectories sampled from the student's own policy, a richer training signal...
arXiv:2610.03176v1 Announce Type: new Abstract: We study hindsight-guided distillation for rare disease diagnosis on ZebraMap: a 1.5B student is fine-tuned on chain-of-thought traces from a 8B teache...
arXiv:2610.03423v1 Announce Type: new Abstract: Reinforcement learning (RL) for Text-to-3D (T23D) generation requires optimization across multiple quality dimensions such as semantic alignment and te...
The paper investigates how reinforcement learning with verifiable rewards (RLVR) interacts with majority voting over sampled completions, a common test‑time scaling technique. It shows that training a single LoRA adapter with the full RLVR budget can improve single‑sample accuracy but often harms the overall majority‑vote performance because the policy’s errors become increasingly correlated. By splitting the same training budget across multiple adapters—each trained on a disjoint shard, forming an "adapter thicket"—the authors demonstrate that this approach consistently out‑votes the fully trained single adapter and preserves most of the single‑sample gains, especially when the number of votes is large.
The paper introduces Density-Aware Reward Aggregation (DARA), a method that adjusts reward weighting in multi-reward reinforcement learning based on the density of active rewards within rollout batches. By applying an inverse-square-root density correction, DARA gives more weight to less frequently active rewards, enabling faster learning of targeted behaviors. Experiments on tool calling and mathematical reasoning demonstrate that DARA achieves comparable final performance while reducing training steps by up to 26% and 65% respectively.
The study evaluates whether biological reasoning models actually use their biological inputs by testing six models on DNA, protein, and single‑cell tasks. By perturbing one biological input while keeping others fixed, the authors find that many models (e.g., Evo2, ESM3, BioReason, BioReason‑Pro) rely primarily on textual information, with minimal impact from the biological representations. In contrast, models like ChatNT, Prot2Text‑V2, CellWhisperer, and Cell2Sentence‑Scale show greater dependence on their biological inputs, yet overall accuracy gains do not consistently reflect increased biological input contribution.
The paper introduces a new primal–dual algorithm for episodic adversarial linear constrained Markov decision processes (CMDPs) with unknown transitions. It achieves a rate‑optimal ×O(√K) regret and cumulative constraint violation, improving upon the previous ×O(K^{3/4}) bound and eliminating the need for Slater’s condition. The method combines adaptive FTRL, contracted value estimation, and an exponential Lyapunov function, enabling uniform concentration over the value function class and computational efficiency independent of the state‑space size.
The paper introduces Adaptive Policy-Guided Error Mitigation (APGEM), a context-aware layer that dynamically selects error mitigation strategies—such as ZNE, PEC, CDR, and REM—during quantum reinforcement learning (QRL) training on NISQ devices. APGEM uses policy-level indicators (quantum-state fidelity, policy entropy, cumulative reward, and approximation ratio) to choose the most suitable mitigation method and integrates it directly into the reinforcement learning loop. Evaluated on the Capacitated Vehicle Routing Problem under various NISQ noise models, APGEM outperforms static mitigation techniques, achieving about 94% of an oracle strategy’s utility, maintaining higher fidelity as noise increases, and producing more stable learning behavior.
The paper investigates how reusing past samples can improve the sample efficiency of Proximal Policy Optimization (PPO). Two variants, wPPO-U and wPPO-BH, are introduced within a multiple importance weighting framework, each reusing data from recent iterations while preserving core PPO mechanics. The authors derive theoretical policy improvement bounds for both variants and empirically evaluate their impact on continuous control tasks.
MiLoop is a reinforcement‑learning‑based constructive framework for neural combinatorial optimization that propagates selective memory across rollout steps. By fusing current embeddings with historical memory before attention layers and applying adaptive gated updates afterward, it enables a shallow policy to learn dynamic embeddings without external solution labels or search‑space pruning. Experiments on four combinatorial optimization problems show MiLoop consistently generates high‑quality solutions for instances ranging from 100 to 10 million nodes, demonstrating strong generalization.
The paper introduces Forward CIP (F‑CIP), an RL‑native formulation of the Controllable Information Production objective that relies solely on system dynamics, eliminating the need for domain‑specific information variable selection. F‑CIP is proven compatible with reinforcement learning and, when applied to existing algorithms, enables agents to autonomously discover primitive behaviors such as balancing and maintaining controllability. Coupled with a simple forward‑velocity reward, the method yields coordinated gaits like hopping and running without reward engineering.
MASkillBlender is a multi-agent reinforcement learning framework that enables decentralized coordination of multiple humanoid robots for loco-manipulation tasks. It learns a shared high-level policy that blends pre-trained single-humanoid skills, requiring only task-level rewards and no task-specific motion references. The approach includes a permutation-based data augmentation technique that preserves policy-gradient direction, and it has been evaluated on several coordination tasks across two humanoid embodiments, consistently achieving strong performance.
The paper investigates continual reinforcement learning using neuroevolution, comparing evolution strategies (ES) and genetic algorithms (GAs) across diverse environments and network sizes. ES consistently achieves a better balance between stability and plasticity, while GAs are more plastic but forget more. The authors attribute this to ES finding wider neighborhoods in weight space, with overlap between consecutive tasks correlating with the stability-plasticity trade‑off, and note that common RL plasticity issues do not transfer to neuroevolution.
The paper introduces dFlowGRPO, a reinforcement learning framework tailored for discrete flow models (DFMs). It generalizes previous work on diffusion large language models by supporting various probability paths and non-masked source distributions, and formulates denoising as a Markov decision process that leverages transition rates and posterior models. Experiments on the multimodal DFM FUDOKI show that dFlowGRPO outperforms existing GRPO methods on text‑to‑image generation and matches continuous flow models on multimodal understanding tasks.
The paper investigates a phenomenon called forking, a type of generalization failure observed in NanoGPT and similar autoregressive models. Forking manifests as a sharp divergence between training and validation loss at epoch boundaries, especially when models employ over‑encoding n‑gram memory modules and data replay. The authors reproduce the effect in both vanilla NanoGPT and a DeepSeek-style model, attributing it to repeated updates that sharpen seen continuations while suppressing unseen ones, with low‑frequency contexts contributing most to the loss gap.
The paper presents linear programming formulations and strongly polynomial algorithms for robust Markov decision processes (RMDPs) with rational polyhedral state-action rectangular uncertainty in rewards and transitions. By encoding a finite sequence of robust policy-iteration steps, a single LP is constructed whose optimal solutions recover the robust optimal value and all optimal stationary randomized policies. The authors provide a general complexity analysis of robust policy iteration, improving known bounds for α1 and α1∞ RMDPs and establishing new strongly polynomial bounds for general interval, weighted α1, and Wasserstein RMDPs, as well as turn‑based stochastic games with these uncertainty sets.
Faynt is a family of Transformer policies (10M and 75M parameters) that control all 26 characters in Super Smash Bros. Melee from a single checkpoint. After reinforcement learning, the 10M model wins 98.4% of same‑character games against fourteen specialist and multi‑character releases, and defeats a zero‑delay Slippi‑AI model in all 68 evaluated games. The work details architecture, scaling, hyperparameter transfer, supervised pretraining on 840,000 human replays, post‑training curricula, distillation, and efficient inference, and it releases the weights, benchmark suites, and a platform for automated model tournaments.
The paper investigates how teacher signals influence parameter updates in Multi‑Teacher On‑Policy Distillation (MOPD) by analyzing Qwen3‑1.7B and SmolLM3‑3B. It shows that loss averaging, Adam’s first‑moment bias, BF16 rounding, and the choice of averaging rule all shape the gradients and ultimately affect task performance. The study quantifies these effects, revealing, for example, that token‑averaging favors longer responses and that BF16 rounding masks most weight changes.
The paper investigates when intrinsic rewards effectively drive exploration in reinforcement learning. It introduces a formal criterion that evaluates policies based on the counterfactual information they acquire, comparing how well their histories can replace experience from alternative policies. Using a simple environment, the authors show that common intrinsic reward objectives—count-based, prediction-error, empowerment, and information-gain—can lead to Pareto-suboptimal exploration under this criterion, and they propose conditions and a new objective that better align with optimal exploration.