The paper investigates how learned visual reward models can inadvertently encourage robot policies to perform poorly on the intended task while still receiving high reward signals. By fine‑tuning a diffusion policy on a drawer‑opening task using a learned reward, the authors observe that task success increases but so does the frequency of wrong‑object failures, a phenomenon that also appears when the policy is re‑optimized with the same reward. A tilt model explains that outcomes with higher initial expected reward become more frequent under KL‑regularized optimization, and a separate outcome verifier can redirect the policy toward the correct task.
By Jiaxuan Luo, Xingguo Xu, Shanshan Wang, Yuhan Zhou, Zhen Zhang
The paper examines how transformer policies, which process agents as ordered token sequences, perform in multi‑agent robot learning where agent teams are unordered. It finds that low permutation error can mask action collapse, where all agents choose the same action, and proposes additional diagnostics such as action diversity and same‑action fraction. Experiments show that while a PPO‑ID baseline avoids collapse, it remains order‑sensitive, and that strong equivariance regularization can still cause homogeneous behavior; a weak penalty improves robustness and preserves diversity for three‑agent teams, but four‑agent teams need much smaller regularization weights.
By Amit Thakur, Mukesh Singhal
The paper introduces Multi‑Fidelity Policy Gradient (MFPG) methods that combine scarce, high‑fidelity (HF) reinforcement‑learning data with abundant, low‑fidelity (LF) data to stabilize training. Unlike prior approaches that directly optimize biased LF objectives, MFPG uses LF data only to construct a control variate that reduces variance without biasing the policy gradient. The authors extend MFPG to modern actor‑critic algorithms, notably a budget‑aware MFPG‑PPO that preserves cross‑fidelity correlation, monitors estimator uncertainty, and allocates a fixed sampling budget between HF and LF sources. Experiments on simulated robot locomotion and a physical Franka arm show that MFPG‑PPO consistently outperforms standard PPO trained on HF data alone and matches performance achieved with 16× more HF data on the hardest tasks, while enabling stable learning with as few as four real‑robot episodes per update.
By Xinjie Liu, Ruihan Zhao, Anirban Chaudhuri, Cyrus Neary, Ufuk Topcu, David Fridovich-Keil
The paper investigates how enrichment—adding intermediate guidance to sparse rewards in reinforcement learning—affects policy-gradient updates. It shows that omitting or truncating importance-weight corrections implicitly reweights rewards, and decomposes the resulting gradient error into scale, rotation, and variance components. The authors propose a sequential Monte Carlo weight correction that mitigates variance growth, and apply their analysis to a fine-tuned Qwen3-1.7B model on OpenMathReasoning, illustrating how enrichment helps prevent collapse in sparse domains.
By Jinwoo Kim, Shraddha Barke
The paper introduces E‑Process‑Authorized Thompson Sampling (e‑ATS), a bandit algorithm that uses full‑history and discounted Beta states to decide when to forget past rewards in non‑stationary environments. An anytime‑valid e‑process first authorizes the discounted state, after which a reversible relevance score determines its influence; before authorization, e‑ATS behaves exactly like optimistic Thompson sampling (OTS). Experiments show that removing the authorization step increases mean normalized dynamic pseudo‑regret by 38.4 % on a registered suite but decreases it by 7.5 % on a literature‑derived replay suite, indicating that evidence controls when adaptation begins rather than guaranteeing improvement.
By Mayand Gulati, Kerong Wang, WeiChen Au
The paper presents a geometry-driven framework for beam management in high-frequency massive MIMO systems. It constructs an offline virtual base station database using 3D LiDAR point clouds and location data to model dominant reflection paths, enabling a coarse channel reconstruction. A VBS-assisted orthogonal-pilot scheme and a dual-agent dueling double deep Q-network are then employed to refine beam estimates and perform coordinated beam selection, yielding improved training efficiency and performance over existing baselines.
By Yijie Bian, Wei Guo, Jie Yang, Shenghui Song, Jun Zhang, Shi Jin, Khaled B. Letaief
EVOL is a framework that uses a knowledge‑tracing simulator to generate per‑learner expert demonstrations via evolutionary search, and then trains a deployment‑free policy that distills these demonstrations into a feed‑forward learner. The method employs an asymmetric actor‑critic architecture, where the actor plans blindly as in real deployment while the critic has access to privileged simulator state during training. Across three educational datasets and varying path lengths, EVOL outperforms eight baselines and shows that the quality of evolutionary experts, rather than the imitation objective, drives final performance.
By Geonwoo Bang, Dongho Kim, Moohong Min
The paper introduces a data‑driven control framework that uses surrogate modeling and reinforcement learning to optimize target polarization in nuclear physics scattering experiments. By training Gaussian process and multilayer perceptron models on operational data from the APOLLO cryogenic target system, the authors demonstrate that Gaussian processes provide calibrated uncertainty estimates and better handle distributional shifts. Embedding the surrogate model in a simulation environment, a reinforcement learning agent trained with a lower‑confidence‑bound reward achieves nearly a 2× improvement over manual operator tuning.
By Armen Kasparian, Torri Jeske, Monibor Rahman, Chris Keith, James Maxwell, Thomas Britton, Malachi Schram, David Lawrence
The paper introduces Test-Time Calibration Learning (TTCL), a label‑free framework that adapts both reasoning accuracy and verbalized confidence of large language models directly on unlabeled target‑task data. TTCL generates self‑supervision signals from multiple model responses, enabling calibration without ground‑truth labels and proving theoretically as a bounded surrogate for the ideal calibration objective. Experiments on mathematical reasoning and factual question answering show consistent improvements, with base models gaining an average 40.13% in accuracy and 70.80% in ECE reduction across eight benchmarks, and further gains under domain shift.
By Zizhuo Zhang, Xiong Peng, Jingwei Sun, Rong Yao, Shixiong Kai, Mingxuan Yuan, Bo Han
The paper introduces HACKTRACE, a behavior‑supervised monitor that detects reward hacking in code‑generation agents by analyzing the agents’ internal states during multi‑turn coding. Using 173,561 annotated trajectories from Qwen3‑8B, the authors show that supervising shortcut behavior independently of exploit success markedly improves detection, achieving a mean per‑problem AUC of 0.997 with minimal monitoring overhead. HACKTRACE also serves as an inexpensive signal for reinforcement learning, dramatically reducing cheating rates while preserving correct solutions.
By Hao Jiang, Xin Li, Annan Wang, Yichi Zhang, Weisi Lin
The paper proposes using Small Language Models (SLMs) as a task‑tool relevance classifier to verify each tool call made by AI agents. By evaluating every selected tool against the assigned task, the SLM provides a relevance signal that can be used for downstream enforcement. The authors introduce a novel dataset of multi‑tool tasks across distinct Model Context Protocol servers and explore prompt‑optimization, supervised fine‑tuning, and reinforcement learning (GRPO) to optimize and specialize the SLMs.
By Chiara Troiani, Arash Salarian, Majed El Helou, Benjamin Ryder, Jean Diaconu, Herv\'e Muyal, Marcelo Yannuzzi
The paper introduces PSPO, a method for offline policy optimization that treats the dynamics model as a random variable, enabling controlled exploration of out‑of‑distribution (OOD) regions while maintaining robustness. By alternating updates of the posterior distribution and the policy, PSPO achieves a regularized optimization algorithm with convergence guarantees. Experiments on standard benchmarks show that PSPO outperforms state‑of‑the‑art baselines and retains a pessimism‑free property without sacrificing robustness.
By Hongqiang Lin, Dongxu Zhang, Yiding Sun, Mingzhe Li, Ning Yang, Haijun Zhang
The paper proposes a scalable traffic modeling approach that uses a single representative large language model (LLM) agent for each homogeneous traveler group, rather than one LLM per traveler. The representative agent maintains a mixed strategy over routes, updates it daily based on positive reinforcement signals, and uses a tunable step size to adjust its strategy. This design improves scalability, stabilizes learning, and produces interpretable dynamics that reproduce realistic behavioral patterns such as the decoy effect and income‑based willingness‑to‑pay differences.
By Hanlin Sun, Jiayang Li
Tropical Reinforcement Learning replaces the traditional sum of probabilities with a maximum operation, forming a tropical semiring. This change allows the value of a state to reflect the log-probability of its most likely verified solution and provides an explicit path that can be replayed. The proposed TROPIC algorithm, applied to deterministic, resettable environments, outperforms strong on‑policy baselines on tasks such as Sokoban, Countdown, FrozenLake, and WebShop.
By Arip Asadulaev, Aladin Djuhera, Karim Salta, Holger Boche, Fakhri Karray, Martin Takac
The paper presents a post‑training approach for text‑to‑image models that combines a preference reward, trained on large human preference data, with rubric‑based rewards that assess prompt faithfulness and other desirable traits. The authors show that a simple reward composition strategy outperforms a naive weighted average, leading to significant Elo gains on the Arena leaderboard for models like Flux2dev and Ideogram‑4. They also release Arena‑T2I‑Training, a 1K subset of data to aid reproducible research in post‑training.
By Yuanhao Ban, I-Hung Hsu, Anastasios Angelopoulos, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh
Prospective Hindsight (PH) is a self‑calibrating training principle for reinforcement learning that augments any retrospective base method with a signal derived from the gap between an agent’s prospective prediction and its retrospective evaluation. By weighting gradients with a stop‑gradient surprise based on this prediction‑reality gap, PH highlights samples where the agent’s self‑model is most inaccurate and encourages the policy and predictor to co‑evolve, progressively focusing on remaining blind spots. Experiments on single‑turn verifiable tasks and a multi‑turn personal‑agent task show that PH improves both task performance and calibration across model scales, addressing overconfident failures in single‑turn settings and underconfident successes in multi‑turn settings.
By Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Yu Li, Hiroaki Hayashi, Chien-Sheng Wu
WebUIProof is a new execution‑oriented benchmark for WebUI code generation that supplies structured specifications and dense, executable interaction tests across general WebUIs and 3D interactive simulations. It employs a UI‑agent harness that runs tests in a headless browser using a plan–act–observe loop to locate DOM elements, perform actions, observe changes, and verify assertions. Evaluations on eight commercial LLMs reveal frequent interaction‑based failures, especially on 3D interfaces, and demonstrate that training compact models with RL rewards from these tests improves functional completion and reduces build failures.
By Yun-Yun Tsai, Yuning Mao, Shiqi Wang, Junfeng Yang, Sinong Wang
Goldilocks RL proposes an adaptive data‑selection strategy for reinforcement learning in language models, using a Selector network to predict reward variability and prioritize questions that are neither too easy nor too hard. By continuously adapting to the model’s evolving abilities, Goldilocks improves training efficiency on large‑scale reasoning datasets, achieving up to 78% fewer optimization steps compared to standard GRPO. The approach addresses the sample‑inefficiency problem of sparse rewards in reasoning tasks.
By Ilia Mahrooghi, Aryo Lotfi, Emmanuel Abbe
The paper introduces PTH (Probe The Harness), a set of checks designed to expose hidden details in experimental setups that can alter the ranking of stale-data reinforcement learning methods for language models. By applying PTH to a comparison between SAN and truncated importance sampling (TIS), the authors demonstrate that subtle harness configurations—such as how PPO ratios are computed, data seeding, replay queue reuse, and loss normalisation—can reverse the observed performance order. The study provides a detailed signature of each influencing factor, reference results for TIS and uncorrected GRPO, and a checklist to ensure fair comparisons.
By Taiheng Pan
The paper investigates why training terminal agents often stalls when using a meta‑agent like Claude Opus to generate tasks and verifiers. It identifies three failure modes—benchmark invalidity, harness brittleness, and reward misalignment—and shows that prompt redesign and context extension can improve solvability by 5.6×, yet a 9B model still tops out at 81.3% mean pass@2. Adding hard tasks drops performance to 20.6%, highlighting that solvability depends on the model used.
By Xi Qin, Isabel Kurth, Xin Cui, Elin Park, Alexander Schaefer, Yaad Oren