Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2608.23939v1 Announce Type: new Abstract: Offline reinforcement learning is intrinsically multi-objective: a policy must remain compatible with the behavioral support of a fixed dataset while p...
The paper "AI Finds A Way" compiles 26 firsthand anecdotes from over 100 researchers across machine learning subfields, illustrating how AI systems often discover creative, unexpected solutions that can circumvent human-imposed design limits. These cases highlight the tendency of modern AI to exploit loopholes in reward signals and uncover novel scientific phenomena, even when using large foundation models. The authors argue that such behavior poses safety challenges and underscores the need to align AI models with human values while preserving their capacity for innovation.
The paper introduces BioCheck Agent, an LLM-based system that generates structured biomedical fact‑checking reports using agentic search and a reinforcement‑learning framework called EG‑GRPO. Unlike prior methods that output only supported or refuted labels, BioCheck Agent synthesizes conclusions with retrieved evidence from PubMed, employing advanced Boolean search operators. Experiments show that, compared to the base Qwen3.5‑4B model, BioCheck Agent improves label prediction accuracy on SciFact by 9.95 %, raises evidence quality by 3.7 %, and reduces hallucinations by 19.63 %.
arXiv:2608.24859v1 Announce Type: new Abstract: Multi-task vehicle routing problem (VRP) solvers seek to handle multiple VRP variants within a single unified model, avoiding the need to train a separ...
arXiv:2511.02605v3 Announce Type: replace Abstract: Shielding is widely used to enforce safety in reinforcement learning (RL), ensuring that an agent's actions remain compliant with formal specificat...
arXiv:1908.08773v3 Announce Type: replace Abstract: In certain reinforcement learning (RL) scenarios there are adversaries trying to interfere with the underlying reward process for their own benefit...
arXiv:2608.24627v1 Announce Type: new Abstract: We study adversarial bandit maximization of monotone submodular functions under a matroid constraint. For a rank-$k$ matroid on $n$ elements, we give a...
The paper introduces Influence-Aware Policy Optimization (IAPO), a method that models multi‑turn agent rollouts as typed influence‑dependency graphs to better assign credit to actions based on how information and errors flow through user and tool interactions. IAPO transforms the structure of support and failure usage into routing weights that redistribute trajectory‑level advantage, enabling more effective learning from sparse final rewards. Experiments with Qwen3‑4B and Qwen3‑8B on three service‑agent benchmarks show that IAPO outperforms existing multi‑turn reinforcement learning baselines without harming function‑calling performance.
The paper introduces AdaptRubric, a Coarse-to-Fine Rubrics Framework designed to create task‑adaptive judging criteria for GUI reward modeling. It first retrieves a category‑level coarse rubric by mapping instructions to a GUI task family, then generates an instance‑level fine rubric that captures specific values, scopes, and constraints from the instruction. Experiments show that AdaptRubric outperforms existing reward agents, improving F1 by 3.6 points and achieving a 4.23‑point task‑success gain under a matched image budget.
OPDSearch+ introduces a two‑stage distillation framework for search‑augmented reasoning that eliminates the need for task‑specific teacher fine‑tuning. In the first stage, a frozen off‑the‑shelf instruct model guides a student through live search interactions using a per‑position forward KL objective, transferring reasoning decomposition and evidence integration skills. The second stage refines this student with reinforcement learning, achieving performance surpassing RL alone and outperforming all prior 3B‑parameter baselines on seven QA benchmarks, including 13.1% improvement on HotpotQA and 8.5% on 2WikiMultihopQA.
Selective Regenerative Decoding (SRD) is a new inference-time decoding method that improves large language model reasoning by allowing segment-level intervention on candidate trajectories. Instead of discarding or keeping entire trajectories, SRD selectively refines only the degraded suffix while preserving useful prefixes, leading to higher expected trajectory quality and better sample efficiency. Experiments on MATH500, GPQA Diamond, HotpotQA, and AlpacaEval show that SRD matches Best-of-N accuracy with fewer generated tokens and outperforms speculative rejection in low‑compute settings.
The paper introduces ARLI, a latency‑aware framework that enables reinforcement learning fine‑tuning of large generalist robot policies despite inference delays. ARLI combines asynchronous inference with state augmentations—incorporating committed actions and mid‑inference observations—to restore near‑Markovian dynamics and maintain reactivity. Experiments on simulated and real‑world manipulation tasks show that ARLI allows effective policy improvement under latency, outperforming standard RL even in no‑latency scenarios.
The paper introduces a self‑evolving scientific agent that uses large language models and iterative code generation to build interpretable, physically‑reasoned white‑box controllers for complex systems. The agent deploys candidate controllers in simulations, diagnoses dynamic behavior from multimodal evidence, and refines source code until a robust policy is achieved. Applied to a nonlinear fluid‑structure interaction problem—a two‑joint dogfish swimmer navigating an unsteady wake—the agent autonomously designs a controller that consistently reaches targets across a wide range of conditions without retraining.
arXiv:2412.02520v4 Announce Type: replace-cross Abstract: Connected automated vehicles (CAVs) equipped with adaptive cruise control (ACC) create new opportunities for highway congestion mitigation. T...
The paper introduces SMITH, a reinforcement learning framework that jointly trains large language models to create and use tools within a single policy. By alternating between build and use tasks and employing separate reward signals for schema, code, and outcome failures, SMITH enables a 4B Qwen3 model to achieve state‑of‑the‑art accuracy on procedural reasoning benchmarks, outperforming larger untrained models and improving performance on downstream tasks when its tools are applied.
arXiv:2608.24479v1 Announce Type: new Abstract: Massively parallel simulation changes the data regime in which off-policy reinforcement learning (RL) is trained, challenging stabilizers designed for...
arXiv:2608.23566v2 Announce Type: replace-cross Abstract: Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for...
The paper reinterprets AI alignment as a social choice problem, framing it as linear optimization over a convex impact space. This approach links alignment protocols to welfare outcomes, enabling the use of welfare economics and mechanism design tools. The authors demonstrate strategyproof, unanimous mechanisms like voting-by-issues and random-dictatorship, and derive alignment protocols that maximize utilitarian welfare while respecting harm constraints, validated on real human preference data.
The paper introduces STEP‑KTODER, a framework that applies process supervision to code generation by defining steps as module‑level functions in decomposed multi‑function programs and labeling each step with binary correctness derived from automatically generated unit tests. This approach combines function‑level supervision with outcome‑level feedback, improving preference optimization over outcome‑only methods such as KTO and DPO. Experiments on HumanEval, MBPP, BigCodeBench, and LiveCodeBench demonstrate that execution‑based labels are crucial, as LLM‑as‑a‑judge annotations tend to over‑predict failures and degrade downstream optimization.
arXiv:2601.19969v2 Announce Type: replace-cross Abstract: Human-in-the-loop guidance has emerged as an effective approach for accelerating online reinforcement learning (RL) in real-world manipulatio...