Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and when to speak at all. Prior w...
MLExpert is a 35B Mixture-of-Experts agentic model designed to autonomously conduct long‑horizon machine learning engineering (MLE) tasks. It uses a post‑training approach that combines hierarchical supervised fine‑tuning with Windowed Interaction RL to learn from intermediate outcomes in complex experimentation workflows. On the MLE‑Bench Lite benchmark, MLExpert raises the average medal rate from 37.9% to 72.7%, matching the performance of larger frontier models and demonstrating transferable optimization capabilities to other benchmarks such as NatureBench Lite.
The paper introduces DSP: Dexterity from Supervised Play, a framework where a robot learns a play policy from brief human demonstrations via behavior cloning and then autonomously gathers extensive interaction data to train a world model. During inference, the play policy guides model‑predictive control by proposing action sequences evaluated by the learned model, and a self‑play distillation step produces a closed‑loop, goal‑conditioned policy. Experiments on underactuated pushing and in‑hand pose reorientation show DSP outperforms hindsight behavior cloning by 69.6% in success rate and matches reinforcement learning performance while using 490 times less data.
The paper introduces a robust reinforcement learning framework that learns adaptive treatment policies for patients, accounting for uncertainties in biomarker measurements and disease dynamics. By treating the patient as a hidden context, the authors unify generalization and robustness through robust contextual Markov decision processes. Applied to a simulator of metastatic prostate cancer calibrated to clinical data, the resulting policies outperform existing adaptive therapy methods, achieving robustness with minimal performance loss under accurate biomarkers and expected disease evolution, and Pareto-dominating the prior approach across varied scenarios.
AVC-Reward introduces an event‑centric framework for evaluating audio‑video consistency in joint generation models. It separates audio and visual events, assesses them independently, cross‑validates cross‑modal relations, and aggregates results into a three‑level metric covering global narrative harmony, event‑level synchronization, and fine‑grained attribute alignment. The resulting AVC‑Reward model aligns closely with human judgments, offers interpretable failure localization, and enhances consistency when used as a reinforcement‑learning signal.
The paper introduces Subspace Tuning (SST), a fine‑tuning technique that confines weight updates to the bottom singular subspace of model parameters, thereby avoiding changes along principal directions that cause catastrophic forgetting. SST is optimizer‑agnostic, works with existing fine‑tuning recipes, and includes dense gradient estimation and a refresh mechanism. Experiments show that protecting up to 99% of the weight subspace significantly reduces forgetting, improves retention, and maintains strong target‑task performance, especially in large‑model regimes, while also reducing KL drift by 2–4× compared to baseline methods.
Routing Policy Optimization (RPO) introduces stochastic exploration within a diffusion visual generator by treating attention routing as a stochastic action. Instead of perturbing the entire reverse denoising trajectory, RPO samples key attention blocks at selected denoising levels, perturbs the transition velocity, and evaluates the perturbation with a score‑function objective. A two‑rollout estimator provides an advantage signal, and after training the model restores dense attention, yielding higher generation quality and nearly four times the training efficiency compared to Flow‑GRPO‑style methods.
A fixed discount factor in Q-learning imposes a single planning horizon throughout training, despite the differing demands of early exploration and long-term credit assignment. We propose Q($γ_H$), a...
In reinforcement learning, advantage-based value-learning methods, unlike standard Q-learning, share information across actions by decomposing the action-value function into a state-dependent value co...
Reinforcement learning with verifiable rewards (RLVR) math corpora are routinely screened for benchmark overlap, but the reliability and data-loss costs of cleaning policies remain underexamined. We a...
TextualRL is a gradient‑free reinforcement learning framework that updates the textual context guiding large language model agents without altering their parameters. It treats groups of rollouts from the same task as units of critique, contrasting successful and failed runs to generate context patches, and validates these patches using held‑out rewards. Experiments on six benchmarks show that TextualRL boosts test accuracy by 24.4 and 15.3 percentage points for GPT‑5.5 and Qwen3.8‑27B, respectively.
The paper introduces GREAT, a Graph Edge Attention Network that operates directly on edge space to handle asymmetric energy costs in the Electric Vehicle Routing Problem (EVRP). Integrated into a reinforcement learning framework, GREAT jointly optimizes routing and charging decisions, outperforming a strong HGS-based heuristic by up to 8.78% in energy savings and reducing runtime by a factor of 60. It also shows competitive performance against a custom adaptive large neighborhood search and achieves state‑of‑the‑art results on other asymmetric routing benchmarks such as TSP, CVRP, and Orienteering.
The paper introduces GATTACA, a scalable framework that applies deep reinforcement learning to control Boolean network models of biological systems, particularly for cellular reprogramming. It formulates a novel control problem under asynchronous updates and enhances scalability by using pseudo‑attractor states and graph neural networks to encode network structure into the DRL agent’s action‑value function. Experiments on large real‑world biological networks demonstrate the approach’s scalability and effectiveness.
The paper investigates whether large language models (LLMs) can learn from rewards presented in context, a process termed in-context reinforcement learning (ICRL). Experiments across four benchmarks and six models show that while LLMs read reward signals, altering or removing these rewards has little impact on performance, and shuffled or corrupted trajectories are as effective as real ones. These findings suggest that direct ICRL behaves more like in-context learning (ICL) than true inference-time reinforcement learning.
The paper presents a Bayesian deep learning approach for active exploration in model-based reinforcement learning applied to robot manipulation. By improving model calibration and actively selecting transitions that maximize information gain, the method enhances data efficiency during a preliminary exploration phase. Experiments on a realistic robotic arm simulator demonstrate that the approach achieves comparable performance to existing methods while requiring significantly fewer robot executions.
The paper introduces a new red‑teaming framework that treats adversarial dialogue with large language models as a Markov Decision Process, employing hierarchical reinforcement learning to handle sparse rewards and long‑horizon planning. By using a token‑level harm reward, the generative agent learns diverse, multi‑turn attacks that consistently uncover vulnerabilities beyond existing baselines. This approach achieves state‑of‑the‑art results and reframes LLM red‑teaming as a principled, trajectory‑based process.
ECLIPTICA is a framework that makes large language model alignment instruction‑driven and runtime‑controllable, allowing natural‑language alignment instructions to modulate behavior on the fly. It introduces Contrastive Instruction‑Tuned Alignment (CITA), which combines supervised fine‑tuning with contrastive preference optimization anchored to a reference model, ensuring stable policy updates within a shared neighborhood. The authors release an ECLIPTICA benchmark of 3,000 controlled cases and demonstrate that CITA achieves 86.7% instruction‑alignment efficiency on Llama‑3.1‑8B, outperforming DPO, GRPO, and PPO.
The paper introduces Scalable Power Sampling, a training‑free method that sharpens a large language model’s generative distribution by approximating the autoregressive power distribution with a token‑level scaled low‑temperature distribution. This approach eliminates the need for iterative Markov chain Monte Carlo, achieving performance comparable to or better than one‑shot GRPO on math, QA, and code tasks while cutting inference latency by more than tenfold. The method relies solely on the base model’s outputs, requiring no external rewards or verifiers.
The paper introduces Selective Successor Eligibility Traces (S-trace), a critic‑free eligibility trace method designed to improve temporal credit assignment in Reinforcement Learning with Verifiable Rewards (RLVR). S-trace reweights successor importance to recover downstream credit signals under PPO-style clipping and selectively propagates them to historical tokens using entropy-based token selection. Experiments on Qwen3 models show that S-trace yields significant absolute gains in average pass@16 over GRPO and trace-style GRPO, while also producing shorter responses.
The paper proposes a reward‑density principle for language‑model post‑training: sparse sequence‑level rewards are best for exploring and discovering new behaviors in a strong teacher, while dense token‑level teacher supervision is optimal for compressing that behavior into a smaller deployment model. The authors validate this rule with a four‑stage workflow—teacher RL, forward‑KL warmup, on‑policy distillation, and optional post‑bridge student RL—showing that an RL‑improved 8B teacher distilled through a dense bridge outperforms direct GRPO on a 1.7B student across math and code benchmarks. Ablation studies confirm each stage’s contribution, and the approach generalizes to Llama and Qwen models, offering a practical guideline for allocating scarce labeled data.