Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2609. 14959v1 Announce Type: new Abstract: We study decentralized learning of Nash equilibria (NE) in infinite-horizon discounted Markov games under bandit feedback, focusing on Markov $\alpha$-potential games.
The paper introduces SCoRE, an agentic framework for Visual Retrieval-Augmented Generation that explicitly selects and consolidates visual evidence before generating answers. It addresses two key challenges: sparse, scattered evidence and noisy exploration trajectories that obscure reasoning. By maintaining a textual ledger of relevant observations and reloading original images for a logical evidence sequence, SCoRE decouples reasoning from exploration and enforces strict visual grounding, with training that rewards evidence coverage, compactness, and answer correctness.
The paper investigates continuous‑time stochastic control problems with unknown drift and running reward functions, using an exploratory reinforcement learning framework that incorporates relaxed controls and entropy regularization. It develops policy‑iteration algorithms based on probabilistic representations of the optimal value function and its gradient, proving convergence and demonstrating performance through numerical examples. The study also extends to a special case with control‑dependent diffusion, requiring a Hessian representation.
arXiv:2609.13901v1 Announce Type: cross Abstract: Multi-charger scheduling in wireless rechargeable sensor networks must weigh sensor death risk, charger energy, travel cost, return-to-base feasibili...
The paper introduces Contextual Scalarisation Thompson Sampler (CSTS), a multi‑objective contextual bandit algorithm that learns to weight competing objectives based on observed context. It addresses the need for adaptable decision‑making in public media, where goals such as audience reach, cultural values, and operational constraints must be balanced. Experiments on Radio Télévision Suisse data demonstrate that CSTS improves contextual relevance and aligns more closely with expert curation than fixed‑weight or standard bandit methods.
The paper critiques the standard fixed-rule approach for deriving labels from human feedback in human-robot collaboration, showing that human-provided implication labels often differ and improve reward learning. It introduces IMPLIED, a method that starts with fixed-rule implications but learns to infer and revise accepted/rejected action labels over time, outperforming both the fixed rule and LLM baselines on recorded trajectories and a physical pizza‑making study. As a result, IMPLIED reduces preference‑estimation error and yields robot actions that better align with combined reward objectives.
The paper discusses the agent-centric general value function (ACGVF) framework, which allows an agent to decide both which goal to pursue and when to consider a goal finished, beyond merely selecting actions. It notes that ACGVF assumes full observability, while a prior approach used an internal belief state but required externally supplied goals. The note proposes to unify and extend these methods using hierarchical hidden Markov models (HHMMs).
The review explores how control theory, optimal transport, probabilistic inference, non‑equilibrium thermodynamics, and machine learning are interconnected through the optimization of free‑energy‑like functionals under dynamical or statistical constraints. It presents a conceptual thread linking these five fields and illustrates the ideas with applications in reinforcement learning, variational inference, and generative modeling. The article is written for readers without prior familiarity, beginning with physics principles.
The paper demonstrates that a single example from the BBQ fairness benchmark can dramatically improve a model’s performance, raising accuracy from 79.9% to 92.9% with Group Relative Policy Optimization and to 99.0% with one-shot in-context learning. This effect is consistent across different model families and is driven by the model’s reasoning traces, which adopt a category‑agnostic "missing evidence" pattern. The authors argue that BBQ-style multiple‑choice abstention tests capture only a single structural cue and therefore do not guarantee true fairness, calling for broader evaluation suites.
The paper presents a mean‑field framework for studying multi‑agent Q‑learning in networked populations, where agents update stateless Q‑values on a fixed graph while the average behavior of the population feeds back to modify the payoff matrix. A deterministic transport equation for the distribution of Q‑values is derived and coupled with a discrete update for the environmental state, and the model is validated against Monte Carlo simulations on several random graph topologies. Results show that the mean‑field system captures macroscopic cooperation dynamics, that environmental feedback reshapes action‑value ordering, and that the timescale of environmental response critically influences learning outcomes.
The paper introduces CREW, a collaborative multi‑agent reinforcement learning framework that automates the generation of the Related Work Section in research papers. Unlike previous methods that follow a fixed workflow, CREW allows large language model agents to dynamically select actions—Retrieve, Disseminate, Compose, and Critique—guided by a policy trained with Independent Proximal Policy Optimization. Experiments on a standard benchmark show that CREW improves output quality and reduces token usage compared to strong baselines.
The paper introduces a label‑free test‑time training approach for GUI grounding, leveraging confidence patterns in coordinate tokens rather than full‑sequence confidence. It proposes Confidence‑Anchored Learning (CAL) to filter pseudo‑labels and assign distance‑based binary rewards, and extends this to Confidence‑Anchored Negative Learning (CANL) which optimizes solely on negative samples to avoid noisy positives. Experiments show that CANL‑7B achieves 92.1% on ScreenSpot‑V2 and 33.8% on ScreenSpot‑Pro, improving the base model by 8.9%.
The paper introduces TTIQ, a test‑time reinforcement learning framework that improves vision‑language model (VLM) adaptation by explicitly measuring image‑question dependence. By teacher‑forcing responses on the original and ablated image‑question pairs, TTIQ derives token‑level likelihood changes to estimate how much each input contributes, then uses these signals to construct a reward that favors jointly grounded, confident responses. Experiments on eight VQA datasets and various VLM sizes show that TTIQ consistently outperforms prior consensus‑based methods and generalizes across model families and unseen datasets.
The paper introduces Drift-Constrained Optimization (DCO), a framework that treats behavioral drift during fine‑tuning of instruction models as a bounded constraint rather than an uncontrolled side effect. By defining a drift budget, the authors reformulate fine‑tuning as a direction‑selection problem, showing that choosing different update directions can qualitatively change outcomes. Experiments on Qwen3 models demonstrate that carefully selected directions improve scientific reasoning and multilingual translation while preserving reasoning capabilities and general performance.
The paper introduces a Large Language Model–enhanced Multi-Agent Reinforcement Learning framework for optimizing electric vehicle charging, station profitability, and grid stability in public charging systems. By using an LLM to select interpretable features from IoT data and dynamically balance conflicting objectives, the approach unifies grid, EV, and station optimization in a single loop. Experiments show the method outperforms existing baselines, improving market efficiency and cutting training time by more than 70%.
The paper introduces SaLT-DPO, a Segment‑aware Listwise Target DPO method designed to improve safety in Large Reasoning Models (LRMs). It decomposes responses into reasoning and answer segments, scores each segment’s safety, and aligns length‑normalized rewards with soft target distributions over multiple candidates. The approach also incorporates joint safety coherence regularization and utility anchoring to reduce unsafe content while maintaining compliance and reasoning performance.
The paper introduces new evaluation metrics for safe reinforcement learning that go beyond average safety guarantees by examining how often and how severely safety bounds are violated, consistency across tasks and bounds, and the relationship between training-time and final policy behavior. It also proposes a safety tier system for categorizing algorithms and presents empirical safety evaluations on multiple navigation tasks. The authors recommend reporting aggregate metrics, distributional data, and task‑specific results together, and provide an open‑source suite, SafeRLEval, to facilitate reliable safety assessment.
HISPO introduces a segment‑level policy‑optimization technique for reinforcement learning with verifiable rewards, creating entropy‑derived contiguous segments during rollout and applying clipped importance‑sampling correction at this granularity. It bridges the gap between token‑level and sequence‑level corrections, offering a middle‑ground approach for credit assignment in long‑form mathematical reasoning. Evaluated on Qwen3‑1.7B‑Base across six benchmarks, HISPO consistently outperforms or matches the strongest baselines in Pass@8 and Acc@8 metrics, notably improving AIME25 scores over GRPO and GSPO.
The paper introduces an exploration-guided prompt scaffolding framework for multimodal large language models, dynamically adjusting the prompt distribution during reinforcement learning post-training. It uses an Exploration Potential Score (EPS) derived from KL-regularized policy improvement to assess prompt utility without extra overhead, and a teacher model rewrites low-utility prompts to preserve intent while improving informativeness. Experiments on Geo3K, MMK12, MathVision, and MMMU-Pro show consistent performance gains, up to 9.7% in-domain and over 11% on out-of-distribution benchmarks.
The paper introduces SPARC, a reinforcement learning framework that generates crystalline materials while respecting symmetry constraints essential for meaningful physical properties. SPARC is applied to two tasks: optimizing uniaxial dielectric anisotropy, which requires specific crystal classes, and maximizing spectroscopic limited maximum efficiency, a scalar objective that lets the algorithm discover suitable crystallographic motifs. The results demonstrate that symmetry is a foundational requirement for producing robust, realizable functional materials.