Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2510.26672v3 Announce Type: replace-cross Abstract: At the heart of reinforcement learning are actions -- decisions made in response to observations of the environment. Actions are equally fund...
arXiv:2604.00055v2 Announce Type: replace-cross Abstract: Existing robotic foundation policies are trained primarily via large-scale imitation learning. While such models demonstrate strong capabilit...
arXiv:2610.07954v1 Announce Type: new Abstract: Language-based trajectory predictors represent coordinates as discrete tokens and learn auxiliary tasks such as destination and group reasoning. This f...
arXiv:2610.08068v1 Announce Type: new Abstract: Realistic physical interaction is a cornerstone of embodied intelligence, yet collecting paired visual--tactile data remains costly. Visual-to-tactile...
The paper introduces Guidance-Augmented GRPO (GA‑GRPO), a theoretical framework that unifies several external‑guidance methods for reinforcement learning with verifiable rewards (RLVR) used to improve large language model reasoning. GA‑GRPO treats external guidance as a stochastic operator that rewrites the question distribution, yielding a biased on‑policy policy‑gradient estimator whose bias is bounded by the total‑variation guidance divergence. The authors prove convergence rates, derive an optimal guidance‑weight formula, and validate their predictions experimentally on a math‑reasoning benchmark, showing that the optimal‑weight GA‑GRPO outperforms existing methods while saving GPU time.
The paper introduces an adversarial reinforcement learning framework to generate synthetic data for Argument Mining (AM). By jointly training a generator and a discriminator, the system produces structured AM instances that are both accurate and diverse. Experiments show consistent performance gains on three benchmark datasets in both full-data and low-resource scenarios.
The paper critiques the use of pass@$k$ as a sole metric for evaluating reinforcement‑learning post‑training on language models, showing that it ignores diversity in model outputs. Experiments on Qwen2.5‑1.5B‑Instruct trained with Group Relative Policy Optimization (GRPO) and rejection‑sampling fine‑tuning (RFT) reveal that these methods shift token‑level, answer‑level, and unique‑answer diversity in opposite directions, yet pass@$k$ fails to capture these differences. Even when restricting to verifier‑correct completions, diversity gaps persist, and pass@$k$ does not consistently identify the better performing arm on hard problem subsets, indicating that the metric can mask true capability changes.
The paper studies reinforcement learning (RL) for post‑training language models on reasoning tasks, focusing on a hierarchical reward structure where each component becomes relevant only after the previous ones are resolved. It demonstrates that a Transformer‑based actor–critic algorithm that alternates between KL‑regularized policy sampling, critic fitting, and policy updates achieves minimax‑optimal rates in query budget and regularization strength, and is optimal for a fixed number of prompts. In contrast, sampling from a fixed reference distribution, as used in offline reward modeling, only yields a logarithmic regret decay, highlighting the advantage of on‑policy exploration in concentrating on high‑reward regions.
arXiv:2610.07358v1 Announce Type: new Abstract: Post-wildfire debris flows (PFDFs) are destructive sediment-laden hazards triggered when intense rainfall strikes recently burned terrain, destabilizin...
arXiv:2609.33155v2 Announce Type: replace Abstract: Test-time scaling and post-training have improved LLM performance in coding and mathematical reasoning, but their effectiveness for individual stan...
arXiv:2610.07767v1 Announce Type: new Abstract: Reinforcement learning (RL) for post-training large language models (LLMs) incurs substantial computation and memory overhead during rollout generation...
arXiv:2610.08271v1 Announce Type: new Abstract: Classical reinforcement learning (RL) assumes that a reward is observed for every visited state-action pair. However, in real-world applications such a...
arXiv:2610.08402v1 Announce Type: new Abstract: Multi-turn LLM agents often receive sparse task feedback across several interactions, while generating each response token by token. This creates two r...
arXiv:2605.09364v2 Announce Type: replace Abstract: This paper investigates robust representation learning in offline goal-conditioned reinforcement learning (GCRL). Particularly in sparse reward sce...
arXiv:2607.09042v2 Announce Type: replace Abstract: Reinforcement learning is increasingly used to fine-tune vision-language-action (VLA) models, but robot interaction is expensive and learning becom...
The paper introduces CASTLE, a framework for fully decentralized multi‑agent reinforcement learning that uses two offline‑trained world models to provide in‑context guidance to independent agents. A Local Dynamics World Model captures each agent’s trajectory dynamics, while a Semantic‑Social World Model predicts short‑horizon task and social outcomes for candidate actions based on counterfactual simulator rollouts. During online learning, these frozen models supply prediction logits to an independent PPO policy, enabling agents to compare potential actions prospectively rather than relying solely on scalar rewards. CASTLE outperforms existing baselines on several multi‑particle benchmark tasks, achieving the highest mean final scores across 30 seeds.
arXiv:2610.06729v2 Announce Type: replace Abstract: Large language models (LLMs) can generate accurate responses, but these are void of diversity. We attempt to address this for the task of creative...
The paper introduces AMBER, an append‑only memory framework for language‑model agents that interact over long horizons. AMBER lets agents jointly learn to reason, act, and write free‑form memory, guaranteeing retention by construction and enabling end‑to‑end reinforcement learning without extensive curated data. Experiments on WebArena Lite show AMBER outperforms overwrite‑based memory by 4.09 percentage points in average success and improves task completion rates in repeated runs.
Rationale-Guided Policy Optimization (RGPO) is a reinforcement‑learning framework that adaptively uses ground‑truth rationale information to scaffold a language model’s reasoning process. Instead of treating reference solutions as fixed imitation targets, RGPO temporarily incorporates rationales to help the model generate better responses, then reverts to unguided learning with higher‑reward, model‑generated solutions. Experiments in both language‑only and vision‑language tasks show that RGPO consistently outperforms RLVR baselines, with ablation studies confirming that adaptive rationale guidance is a key factor in its success.
arXiv:2610.07556v1 Announce Type: new Abstract: Learned orchestration can automatically construct effective language-model multi-agent systems, but existing approaches couple planning to fixed worker...