Policy optimisation, reward modelling and RLHF — how models are trained by feedback rather than by labels.
arXiv:2609.06586v1 Announce Type: cross Abstract: A shared reward gives agents a common objective, but leaves open when, how and even whether they must cooperate to succeed. We address these question...
arXiv:2609.09876v1 Announce Type: new Abstract: Dense change detection in remote sensing requires vision-language models (VLMs) to compare bi-temporal images and generate accurate pixel-level masks....
arXiv:2609.08064v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed in settings where rare but severe harmful generations can have significant consequences. Existin...
The paper introduces ARC, a lightweight hierarchical policy that learns to configure LLM‑based agent systems on a per‑query basis by treating each configuration as a temporally extended option in a semi‑Markov decision process. Unlike fixed templates or hand‑tuned heuristics, ARC dynamically selects workflows, tools, token budgets, and prompts tailored to the difficulty of each query. Experiments on reasoning, tool‑use, and agentic benchmarks show that ARC outperforms budget‑matched tool‑augmented LLMs, boosting reasoning accuracy by 31.3%, tool‑use accuracy by 13.95%, and doubling success on the τ‑Bench Airline Pass task from 9.0% to 18.0%.
FrogNano is a 4B coding agent trained exclusively with reinforcement learning on about 1,500 synthetic software engineering environments. Its training leverages an online task synthesis pipeline that generates tasks at the current agent’s learnability frontier, improving performance without distilling from larger models. The report details the methodology, evaluates the agent across diverse environments, and analyzes its effectiveness as a lightweight coding agent for minimal hardware.
arXiv:2609.07756v1 Announce Type: new Abstract: Inertial confinement fusion (ICF) is a leading pathway toward clean energy, but each shot at the National Ignition Facility costs on the order of one m...
The paper investigates how reinforcement‑learning policies for legged robots encode gait information by examining the effective rank of the policy Jacobian conditioned on gait phase. It finds that common architectural features such as layer normalization and residual connections allocate more representational capacity to swing than stance, a pattern absent in vanilla MLPs. Leveraging these insights, the authors propose a simple recipe that improves sim‑to‑real transfer, reducing joint jitter on a physical Spot robot by roughly three‑fold.
Spectral Prioritized Sweeping (SPS) extends traditional Prioritized Sweeping by incorporating graph topology through the resolvent and Laplacian diffusion, creating a smoother priority score that propagates reward changes more effectively in nonstationary reinforcement learning. The method, called Graph Topology Augmentation for Prioritized Sweeping (GTA-PS), uses a mixing of regularized Laplacian inverses and an adaptive scheduler based on the Second Largest Eigenvalue Modulus to adjust the influence of topology during replanning. Experiments on FourRooms and GARNET domains show that GTA-PS improves replanning efficiency compared to standard PS under both exact dynamic programming and Dyna-style planners.
Bakhtiari, Lattimore and Szepesvári (COLT 2025) proved that Thompson sampling (TS) has Bayesian regret $\tilde O(d^{5/2}\sqrt n)$ for bandit convex optimisation with convex \emph{monotone} ridge losse...
While Large Language Models (LLMs) have demonstrated impressive capabilities, they often struggle with extremely long contexts due to fixed context limits. To address this, sequential approaches like...
Finding new physics (NP) is the most important problem in particle physics today. Studying ``anomalies'', i.e., measurements of low-energy observables whose values disagree with the predictions of the...
Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke. We show that this recipe becomes structurally mismatched in specialist sc...
Asynchronous reinforcement learning has become the standard way to scale training for language models, but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on...
The paper presents a reinforcement learning method, HSAC, that builds covering structures without relying on rigid, pre‑planned sequences. It uses graph‑structured state representations and a mixed action space to select blocks and adjust their placement continuously, while an efficient exploration strategy incorporates unilateral edges into graph neural networks. HSAC outperforms the prior hybrid‑PPO approach, shows strong sample efficiency, robustness to hyperparameters, and successfully transfers policies from simulation to a real two‑robot 3D‑printed block construction task.
The paper introduces SUN, a reachability-aware goal-selection framework for reinforcement learning that jointly considers novelty and reachability. SUN uses successor value functions to identify goals that are both novel and reachable, proving properties such as recovering count-based bonuses, bounding short-horizon hitting probabilities, and rejecting unreachable goals. An adaptive goal-selection strategy and a lightweight pseudocount are proposed, and extensive benchmarks show SUN outperforming state-of-the-art methods across diverse environments.
The study investigates how different types of caregiver feedback influence grammar learning by training small GPT‑2‑style models on child‑directed language and fine‑tuning them with reinforcement learning. Four feedback categories—communicative, structural alignment, semantic contingency, and affective—were evaluated, with structural alignment showing the strongest improvement in grammaticality and communicative feedback yielding moderate gains. Semantic contingency and affective feedback did not enhance grammaticality, though they may aid other language learning aspects, indicating that various feedback forms contribute complementarily to language acquisition.
The paper introduces a history‑aware offline reinforcement learning policy that predicts iterative cost weights for routing in dense integrated circuit designs. By incorporating a lightweight LSTM and additional router features, the policy retains sequence context and improves convergence across various placement densities and guide qualities. Integrated into any cost‑based router with minimal changes, the approach reduces design rule violations by an average of 92% and cuts runtime by 10%.
Inertial confinement fusion (ICF) is a leading pathway toward clean energy, but each shot at the National Ignition Facility costs on the order of one million dollars, making accurate AI surrogates a h...
ConsensusBench is a new dataset that supplies rule‑based process‑level signals for large language model reasoning. It identifies key intermediate conclusions—called Consensus Nodes—by filtering correct trajectories and clustering semantically equivalent statements. By incorporating a process reward derived from these nodes into GRPO‑style reinforcement learning, the authors create ConsensusPR, which reduces reward sparsity and improves performance on benchmarks such as AIME, GSM8K, and MATH‑500.
VLA-Precision introduces an efficient real‑world online reinforcement learning framework for vision‑language‑action (VLA) models, featuring the Asymmetric Co‑Bootstrapping (ACoB) algorithm and the ACoB‑Stream architecture. ACoB uses asymmetric co‑bootstrapping across timescales to rapidly improve policy performance while refining value estimates, thereby reducing policy drift. ACoB‑Stream enables large VLA models to run with up to 10.9× higher throughput and computational efficiency, achieving a 98.3 % mean success rate on nine high‑precision chemistry tasks in under 46 minutes per task.