Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2608. 12973v3 Announce Type: replace-cross Abstract: In this paper, we study online statistical inference for quantile-based distributional policy evaluation, which aims to characterize the distribution of discounted cumulative rewards under a given policy.
RoutePack is a replay‑guided planner that coordinates expert placement and rollout‑trajectory scheduling for Mixture‑of‑Experts reinforcement learning models. By incrementally adjusting expert placement while preserving residency and then rescheduling trajectories into token‑capped micro‑batches, it jointly balances dense‑side work and layer‑wise expert peaks under fixed capacity constraints. Evaluations on 7.9B and 124B hybrid‑attention MoE models across NVIDIA H20‑3e configurations show throughput gains of 4.44‑11.85% over a DP‑local first‑fit‑decreasing baseline and 1.33‑5.28% over expert rerouting alone.
The paper introduces Grounded Credit Assignment (GCA) for reinforcement learning in multi‑turn search agents. GCA separates credit for evidence retrieval and answer correctness, correcting the bias in GRPO that favors evidence‑retrieving rollouts only proportionally to accuracy gains. Applying GCA to Search‑R1 boosts average EM by 5.7–6.3 points and increases the proportion of evidence‑supported correct answers from 76.9% to 93.4%.
Process Reward Models (PRMs) are crucial for scaling large language model reasoning at test time, but they suffer from two types of uncertainty: PRM uncertainty from limited supervision and imperfect training, and policy uncertainty from mismatched data collection policies. The paper introduces BayesianPRM, which treats process rewards as random variables, discretizes the reward space with an ensemble, and applies two belief updates—reliability‑aware updating for PRM uncertainty and conservatism‑aware calibration for policy uncertainty—to form a Bayesian belief whose expectation yields a more accurate reward estimate. Experiments on multimodal reasoning benchmarks show that BayesianPRM improves reward accuracy and test‑time scaling performance.
DiffusionRVGE presents a reinforcement learning approach that improves diffusion models by estimating value gradients through regression of centered rewards on centered final denoised images, avoiding costly reward derivatives and backpropagation through the sampling trajectory. The method adapts rollout bandwidth and query weights based on reward continuity, enabling efficient training across multiple rewards. Experiments show that a single SD3.5‑Medium model trained with DiffusionRVGE matches or surpasses DiffusionNFT on most metrics while using roughly one‑fifth of the training compute, and achieves faster reward convergence in single‑reward settings.
The paper introduces a cost‑aware controller called certified predictive advice gating that decides when to request language‑model advice in reinforcement learning. By forecasting advisor responses, evaluating parsed actions, and reusing cached advice, the gate only requests advice when a calibrated lower bound on the value exceeds the cost. Experiments on BabyAI tasks show that the gate improves return while reducing advice calls by over 97 % compared to always asking, though matched‑budget comparisons do not show a clear advantage over an early‑query schedule.
CastFlow is a role‑specialized agentic forecasting framework that combines general‑purpose tool reasoning with specialized temporal forecasting. It uses an Experience Memory to guide a frozen LLM in acquiring evidence via a multi‑view toolkit, then a lightweight specialized LLM refines predictions through evidence‑grounded reasoning. The two‑stage training strategy—supervised fine‑tuning followed by reinforcement learning with verifiable rewards—optimizes the specialized model based on final forecasting outcomes, achieving superior performance on nine out of ten real‑world benchmarks.
The paper introduces EquivAug, a data augmentation framework that applies symbolic equivalence transformations to chain-of-thought (CoT) solutions, generating diverse yet provably correct reasoning paths for supervised fine-tuning (SFT). By converting CoT solutions into formal symbolic representations, applying equivalence transformations, and reconstructing natural-language explanations, EquivAug produces training data that varies in reasoning steps rather than surface-level lexical differences. Experiments on GSM8K and FOLIO show that EquivAug consistently achieves the highest reasoning path diversity after SFT and the strongest accuracy after GRPO, establishing a positive correlation between CoT diversity during SFT and downstream reinforcement learning with verifiable rewards (RLVR) performance.
NestedVLA proposes a new approach for consolidating skills in Vision‑Language‑Action models by learning skills independently and then generating task‑specific parameters via a nested hypernetwork, rather than aggregating parameters. The method keeps a frozen base VLA and task‑specific compressed VLM residuals, enabling flexible, interference‑mitigated skill reuse. Experiments on MetaWorld, ManiSkill, and CALVIN show that NestedVLA outperforms the best model‑merging baseline by 41 pp on average and approaches per‑task fine‑tuning performance.
The study investigates whether five large language‑model agents adapt their contributions in a collective‑risk dilemma game. Across ten rounds, most models maintained a roughly fixed contribution level regardless of changing risk or partner behavior, and many continued to contribute even when it offered no payoff advantage. The results reveal that cooperative self‑play does not necessarily indicate strategic adaptability, and that agents’ responses to incentives, partner actions, and population composition vary significantly.
ECHOES is a method for long‑horizon GUI agents that compresses visual history into a textual state using a Compression‑Think‑Action format. It employs a privileged teacher model that provides token‑level on‑policy distillation to guide the student’s textual rollouts, while a Perception‑Memory Fidelity Gap‑guided framework balances distillation with task reward optimization. The approach is evaluated on four GUI benchmarks and a new ECHOESBMK diagnostic benchmark, showing improved task performance, memory fidelity, and a favorable trade‑off between performance and inference cost, with positive zero‑shot transfer to unseen planning tasks.
GlobeBench is a new benchmark for evaluating text-based world models, comprising 1,632 tasks across five domains and nineteen simulation skills. It tests whether models can predict an environment’s response to an action based on the full agent trajectory, requiring understanding of state changes, environment rules, and action effects on related and unrelated objects. The benchmark compares predictions to execution-derived references using a hybrid architecture of deterministic gates and a semantic judge, and finds that GPT‑6 Astra tops the 20 tested models with a 49.57% success rate, while accuracy drops sharply when actions affect many dependent objects, leaving 631 tasks unsolved by any model.
R$^2$Orch introduces a Recursive Self-Play Orchestration framework that uses an inner self‑play loop to improve an orchestrator on self‑selected tasks and an outer loop that refines how the inner loop improves. The method employs Recursive Learnability to reward task selection that yields diverse outcomes as skills are removed, and Skill Withdrawal to prune skills when the orchestrator performs equally well without them. Experiments on twelve datasets demonstrate that R$^2$Orch outperforms existing baselines in question answering, math, code, and interactive tasks.
Reward Engine is a method for creating a reward function for open‑ended tool agents by first determining what success looks like and then optimizing toward it. It builds a RewardPack of test cases and rubrics from initial trajectories, which is frozen and used to guide the agent’s output selection and can also serve as a reinforcement‑learning reward. Experiments on WorkBuddy Bench Office and Claw‑Eval show that the constructed reward aligns with benchmark graders on most candidate pairs and improves three‑run consistency from 45.0% to 54.0% and from 78.4% to 83.7%.
The paper introduces Distill Until Divergence, a method for agentic reinforcement learning that uses error-truncated hindsight distillation. It routes two complementary hindsight skills—Positive for successful workflows and Negative for failure corrections—based on trajectory outcomes and error boundaries. The post-error segment is optimized solely by GRPO, preserving outcome-based learning while avoiding mismatched hindsight supervision.
The paper introduces Agentic In-Context Learning (Agentic ICL), a framework where fixed‑parameter Transformers learn from interactions that shape the context they receive. It models an interactive learner with state, update, and decision rules, covering active learning, bandits, and online reinforcement learning, and demonstrates that causal Transformers can approximate these algorithms over a finite horizon. Experiments show that Transformers can learn to execute recursions and that different reward signals produce distinct decision policies, with alignment between process supervision and outcome‑optimal actions influencing learned behavior.
WebCoder is a general frontend development agent that uses artifact‑grounded agentic reinforcement learning to learn how to compose, generate, modify, and repair code over an evolving application state. It evaluates development actions by their impact on a shared executable artifact and a verifier that provides terminal rewards, while intermediate progress redistributes rollout advantage across development segments. The system includes Webdev‑World, an executable environment that integrates code editing, execution, browser inspection, and on‑demand skills, and Webdev‑Bench, a benchmark for composed development tasks, where WebCoder‑27B outperforms several large language models on requirement satisfaction and visual/functional scores.
AlphaBet is a reinforcement learning framework that trains a single large language model to search for evidence, reason about it, and report probabilities for unresolved events across multiple pre-resolution dates. By scoring each dated forecast with a proper scoring rule, AlphaBet uses one resolved event to supervise decisions under varying information states, decomposing forecasting regret into evidence-acquisition and probability-reporting gaps. Experiments on GLM-4.5-Air show that AlphaBet improves Brier scores on Forecast-Dojo and FutureX, attributing gains to multi-date training and better information acquisition and probability reporting.
arXiv:2610. 22887v1 Announce Type: cross Abstract: Long rollout windows make reinforcement learning with verifiable rewards (RLVR) expensive, yet simply shortening them can remove the correctness signal: when every single-problem rollout truncates before an answer, group-relative accuracy advantages vanish.
The paper introduces AnchorQ, a method that improves sample-efficient reinforcement learning for policy refinement by using flow-reversed action anchors. AnchorQ maps reference behaviors to state-conditioned latent anchors, samples local particles around these anchors, and employs value-guided sampling to explore promising refinements. Experiments on simulated and real-world manipulation tasks show that AnchorQ outperforms state-of-the-art post-training baselines, achieving a 37% higher real-world success rate and 97% greater simulated task throughput within the same interaction budget.