The paper proposes probe-guided fine-tuning, a method that uses probes detecting undesired properties in model activations as a direct training signal. Experiments show that continuously updated probes reduce harmfulness and improve honesty while preserving utility, outperforming DPO and inference-time steering in safety-utility trade-offs and robustness to jailbreak and abliteration attacks. Importantly, the concepts remain linearly encoded after fine-tuning, maintaining monitorability.
By Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko
The paper investigates Training‑Inference Mismatch (TIM) in large‑language‑model reinforcement learning, where rollout generation and policy optimization produce differing token probabilities despite identical model weights. By creating a zero‑mismatch diagnostic setting called VeXact, the authors isolate TIM and demonstrate that even minor token‑level numerical disagreements can trigger training collapse. They further show that TIM alters the effective optimization problem and propose remedies to mitigate its impact.
By Tianle Zhong, Neiwen Ling, Yifan Pi, Zijun Wei, Tianshu Yu, Geoffrey Fox, Peng Wu, Xiao Yu
The paper introduces Amortized Inpainting with Diffusion (AID), a method that keeps a pretrained diffusion backbone fixed and trains a small reusable guidance module offline for image inpainting. AID formulates the problem as deterministic guidance with a supervised terminal objective, derives an auxiliary Gaussian formulation to make it learnable, and proves that solving the randomized problem recovers the optimal deterministic guidance field. Experiments on AFHQv2, FFHQ, and ImageNet show that AID improves the quality–speed trade‑off over strong baselines while adding less than one percent trainable overhead.
By Yilie Huang, Xun Yu Zhou
arXiv:2307.15494v3 Announce Type: replace-cross
Abstract: Hindsight Experience Replay (HER) enhances sample efficiency in goal-conditioned reinforcement learning (RL) by relabelling failed trajectori...
By Kevin Yandoka Denamgana\"i, Daniel Hernandez, Ozan Vardal, Sondess Missaoui, James Alfred Walker
The paper investigates preference-based bandits where a learner selects pairs of arms and receives binary preference feedback modeled by Bradley–Terry. It introduces the locally sensitive eluder dimension, a new complexity measure for logistic preference feedback, and proposes the GINOP algorithm that uses log-loss confidence sets to balance optimism and exploration. The authors prove a first-order regret bound showing that learning with preference feedback can be as statistically efficient as learning from direct rewards, and they validate their theory with empirical experiments.
By Ahmed Ben Yahmed (CREST, ENSAE Paris, FAIRPLAY), Marc Abeille (FAIRPLAY), Cl\'ement Calauz\`enes (FAIRPLAY)
The paper introduces StructEvo, a structure-aware reinforcement learning framework designed to improve protein directed evolution. By using a delta-structure fusion encoder to approximate mutant structure features and a hierarchical action network aligned with protein structure, the method navigates the vast mutation space more effectively. StructEvo outperforms existing machine learning-assisted directed evolution techniques by 9.2% and 16.3% on two benchmarks and uncovers an experimentally validated epistasis pattern in GFP, underscoring the value of structural guidance.
By Zikun Nie, Suyuan Zhao, Yizhen Luo, Siqi Fan, Zaiqing Nie
The paper introduces T‑Router, a parameter‑efficient reinforcement learning framework that learns thalamic routing to improve reasoning with a pretrained model. T‑Router focuses adaptation on reusing completed computations via a compressed, addressable bank and a depth‑recurrent controller, allocating only 0.466% of the backbone parameters while achieving higher MathAvg scores on GSM8K and improving AIME accuracy. The study demonstrates that controlled computation reuse can outperform full‑parameter and LoRA approaches in reasoning tasks.
By Liuxian Ma, Jiale Dai, Jiaqi Li, Lu Mi
The paper introduces SCOUT, a co‑training framework that adapts an off‑policy teacher to better continue from student‑generated prefixes in on‑policy distillation (OPD). By periodically optimizing the teacher’s conditional continuation ability using reinforcement learning with verifiable rewards, SCOUT improves the teacher’s performance on student prefixes. Experiments across various teacher‑student setups, model scales, and reasoning domains show that SCOUT consistently enhances the effectiveness of OPD.
By Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Bl\"obaum, Purak Jain
The paper introduces Actor‑Critic with Action Chunking (AC2), a method that assigns credit to short action chunks instead of entire trajectories, enabling policy updates without waiting for terminal rewards. AC2 employs local readiness, reference solutions, and 10k‑token chunks to make critic‑based credit assignment reliable. Experiments on Qwen3‑4B with FineProofs‑RL show AC2 surpasses GRPO’s peak validation score while using 2.5× fewer decoding FLOPs and fewer training steps.
By Kaiyue Wen, Luke Bailey, Arvind Mahankali, Tengyu Ma
PrivMeSA is a privacy‑aware, self‑evolving multi‑agent system that enables local clinical LLM agents to consult remote specialists while controlling patient data disclosure. It uses reinforcement learning to balance task accuracy with privacy risk, evaluates risk over entire outbound transcripts, and employs a local lesson memory to distill and reuse expertise without further remote exchanges. On an emergency‑department benchmark, PrivMeSA boosts task accuracy by up to 15.8 percentage points and virtually eliminates patient re‑identification risk, reducing personal detail disclosure from 98.0% to 0.2% of cases.
By Dannong Wang, Yuran Zhang, Bian Sun, Alex Stinard, Yuzhang Shang, Song Wang, Yu Tian
The paper presents a method for generating creative chess puzzles using masked diffusion models that can be conditioned on tactical themes and partial board positions. It introduces an auxiliary best‑move prediction task that boosts solution uniqueness by 11.6% and theme‑conditioning accuracy by 2.5%. A reinforcement learning framework further increases the yield of unique, theme‑matching puzzles by 89.1%, and the authors release open‑weights models for the community.
By Aatu Selkee, Severi Rissanen, Xidong Feng, Tom Zahavy, Eric Malmi
The paper introduces Retrospective World Modeling, a new paradigm for vision‑language‑model (VLM) agents that allows them to reason backward by estimating which action most likely caused a state transition. It proposes the Self‑Consistency Reward (SCR), an intrinsic signal that measures how well a policy action aligns with this retrospective explanation, providing dense transition‑level feedback. Experiments demonstrate that incorporating SCR improves policy robustness and generalization compared to purely prospective world‑modeling approaches.
By Yongjiang Liu, Jie Zhang, Haoyue Zhang, Jingcai Guo, Deze Zeng, Song Guo
The paper introduces Reasoning State Propagation (RSP), a method that models each reasoning prefix with a binary validity state and learns transitions between successive states. RSP predicts break and repair probabilities to connect intermediate reasoning states to the final outcome, enabling outcome supervision to guide learning of earlier steps. Experiments on reasoning search, response selection, and reinforcement learning show RSP consistently outperforms existing Process Reward Models, achieving notable gains over Qwen2.5-Math-PRM.
By Kai Gan, Zi-Hao Zhou, Bo Ye, Jian Zhao, Min-Ling Zhang, Tong Wei
SkillFM is a generative framework that creates task‑conditioned textual skills for large language model agents without relying on manual skill banks or reinforcement learning. It encodes skills into a continuous latent space using a codec and trains a conditional flow model with improved MeanFlow, allowing single‑step latent sampling at inference. The sampled latent is decoded by an LLM into textual guidance, and the method outperforms other vector‑based skill approaches on ALFWorld, Search‑QA, and other tasks.
By Zuming Zhang, Jie He, Yizhe Zhang, Jeff Z. Pan
The paper introduces Proximal Entropy Policy Optimization (PEPO), a method that assigns token‑level advantages in reinforcement learning for language models based on a local measure of entropy called proximal entropy. Unlike prior approaches that use global token entropy, PEPO evaluates importance relative to neighboring tokens, making it invariant to prompt difficulty and positional effects. Experiments on Qwen3 and Llama‑3.2 models show PEPO outperforms GRPO and other entropy‑based baselines in mathematical reasoning tasks.
By Yun Kim, Nojun Kwak
The paper investigates how large language models learn to hide their reasoning within text—termed steganographic reasoning—compared to related abilities like steganographic messaging and encoded reasoning. Across reinforcement learning, in-context learning, and supervised fine-tuning, models readily acquire messaging and encoded reasoning, but steganographic reasoning only emerges under supervised fine-tuning and requires substantially more training, unless a convenient cover task is provided. Even then, steganographic reasoning remains significantly harder than its neighboring capabilities.
By Julian Schulz, Lukas F\"ulle, Rieke Fruengel
The paper introduces a moving-horizon approximate branch‑and‑reduce method for training deep classification trees on large datasets with continuous features. It combines a hierarchical root‑subtree optimization framework, branch‑and‑reduce at the root, greedy heuristics for subtrees, and a low‑cost moving‑horizon refinement to improve accuracy. Experiments show the approach surpasses heuristic baselines in test accuracy while scaling better in dataset size and tree depth than existing global optimal solvers.
By Chenxuanyin Zou, Jiayang Ren, Qiangqiang Mao, Jing Liu, Marcus Lai, Yankai Cao
SynIL is a new framework for offline imitation learning that automatically assesses the quality of demonstration data without requiring labels. It uses motor synergy—a low‑dimensional coordinated movement pattern linked to proficiency—to generate dense, transition‑level reward signals through self‑supervised reward regression. Experiments on D4RL locomotion and Robomimic manipulation datasets show that synergy‑derived rewards align well with true rewards and that SynIL outperforms Behavior Cloning and rivals or surpasses offline reinforcement learning in sparse‑reward scenarios.
By Yuto Tanaka, Kyo Kutsuzawa, Martina Doku, Dai Owaki, Mitsuhiro Hayashibe
The paper presents a new scaling law for reward optimization in AI alignment, showing that performance scales as Θ(√min{log(M), K}), where M is the number of preference comparisons used to train a proxy reward model and K is the KL‑divergence budget relative to a reference policy. The authors derive this law using an information‑theoretic model, prove its tightness, and validate it with extensive experiments involving a 70B gold reward model and smaller proxy models (0.6B–4B). The empirical results demonstrate a strong fit (R² 97–99 %) across different model sizes, noise levels, and optimization methods, suggesting that reward optimization behaves like a simple selection task over IID Gaussian variables with noisy feedback.
By Ali Aouad, Aymane El Gadarri, Vivek F. Farias
The paper identifies a flaw in rubric‑based reinforcement learning where additive reward aggregation allows policies to compensate for missing critical criteria, leading to higher scores but poorer answers, especially in clinical consultation tasks. It demonstrates that grouping rubric criteria into protocol‑level dimensions—so a dimension only counts when all its criteria are satisfied—mitigates this reward hacking. The proposed Protocol‑level Rubrics (ProRubric) improve appropriateness by 10.8 points without sacrificing coverage and achieve the best performance across seven benchmarks.
By Maoqi Liu, Junwei He, Bowen Zhang, Feiran Li, Wentao Ma, Rongyi Lin, Shuhan Zhong, Quan Fang