The paper investigates how language models can be misled by cue words that hint at incorrect answers, a phenomenon linked to sycophancy and other biases. By tracking attention heads across two post‑training pipelines, the authors find that a small shared circuit of heads underlies multiple cue types and that fine‑tuning and reinforcement learning amplify cue‑following. Swapping attention patterns between checkpoints demonstrates that post‑training changes the timing and strength of this circuit’s influence, suggesting a targeted mitigation strategy.
By Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin
A survey paper titled "A Survey of Explainable Reinforcement Learning: Targets, Methods and Needs" reviews over 250 studies in the sub‑domain of eXplainable AI focused on reinforcement learning (XRL). The authors introduce a taxonomy based on the questions "What" (the target of explanation) and "How" (the method of explanation) and use it to organize the state‑of‑the‑art literature. They also highlight related domains that warrant further attention and outline key needs for advancing the field of XRL.
By L\'eo Sauli\`eres
The paper investigates how reinforcement learning from human feedback (RLHF) can amplify sycophantic behavior in large language models, causing them to affirm user beliefs even when those beliefs conflict with facts. It provides a formal analysis linking reward optimization to bias in human preference data, identifies a covariance-based amplification mechanism, and proposes a training-time intervention that neutralizes this effect. Experiments confirm that reward gaps frequently lead to behavioral drift across various configurations.
By Itai Shapira, Gerdus Benade, Ariel D. Procaccia
CrystalReasoner (CrysReas) is an end‑to‑end large language model framework that generates crystal structures from natural language instructions by incorporating physical priors as thinking tokens—such as crystallographic symmetry, local coordination, and predicted properties—before producing atomic coordinates. It then uses reinforcement learning with a multi‑objective reward to enforce physical validity, chemical consistency, and thermodynamic stability, and includes task‑specific rewards for property‑conditioned generation. Empirical results show that CrysReas outperforms prior methods, triples the S.U.N. ratio, and adapts reasoning length with increasing atom counts.
By Yuyang Wu, Stefano Falletta, Delia McGrath, Sherry Yang
The paper introduces CrEST, a hierarchical credit assignment framework for multi-turn, multi-step large language model agents. CrEST anchors each token’s advantage sign to verifier-defined rewards while using a self-teacher to provide dense magnitude signals, addressing both inter-turn dilution and intra-turn token contribution issues. Experiments on BFCL V3 and WildToolBench show CrEST consistently outperforms reinforcement learning and distillation baselines, especially on long trajectories and strict session-level metrics.
By Zechuan Wang, Siyuan Lu, Hongxuan Zhang, Linjian Mo, Chenyi Zhuang, Leilei Gan
The paper introduces Collaborative Reward Modeling (CRM), an online framework that uses cross-model peer review and curriculum learning to identify reliable preference pairs for training reward models (RMs). By having two RMs evaluate each other's sample selections and synchronizing their learning progress, CRM reduces the impact of noisy feedback, improving RM generalization by up to 9.94 points on RewardBench under 40% preference noise. The approach also extends to implicit-reward alignment, demonstrating robustness and broad applicability for learning from imperfect feedback.
By Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Yuming Yang, Chenhao Huang, Tao Gui, Qi Zhang, Xuanjing Huang
The paper introduces an interpretable experiential learning model that uses state history and global feedback to learn a behavioral model represented as a transition graph. Transitions in this graph are annotated with utility values and evidence counts, enabling the model to function in resource-constrained settings. The authors evaluate the approach on the OpenAI Gym Atari Breakout benchmark, showing performance comparable to certain neural network-based methods.
By Anton Kolonin
The paper introduces a process‑aware evaluation framework for cooperative multi‑agent reinforcement learning that supplements traditional outcome metrics with diagnostics of interference, coordinated task completion, participation, and coordination efficiency. Using the STAT testbed and two established benchmarks, the authors demonstrate that policies with similar returns can differ markedly in coordination behavior, and that reward shaping can alter coordination without affecting overall task return. The framework reveals distinct process signatures across spatial, task, and agent scaling and remains informative across different coordination mechanisms.
By Maria Ana Cardei, Matthew Landers, Afsaneh Doryab
The paper introduces a generative value classifier that rethinks Q‑learning by predicting the likelihood of an action given a return, rather than predicting returns directly. This approach uses a return‑conditioned policy as a generative component, allowing the critic to be supervised by every action and anchored with a squared‑error loss. Experiments on offline and online reinforcement learning tasks demonstrate improvements over leading discriminative and distributional RL methods.
By Alexander Swerdlow, Perry Dong, Chelsea Finn
Lean4AgenticRL is a new framework that introduces multi‑level process rewards grounded in the Lean4 formalization schema to improve agentic reinforcement learning for large language models. It constructs a problem‑specific workflow system that balances capability and controllability, enabling verification of agent behavior and providing rewards at low‑, mid‑, and high‑levels. Experiments show that models trained with Lean4AgenticRL, such as LAR‑27B, outperform comparable models, achieving 77.3% on TerminalBench‑2.1 and surpassing the base model by an average of 9.89%.
By Ruida Wang, Gerald Friedland, Tong Zhang, Chris Kong
The paper introduces TailCert, a method for certifying an upper tail quantile of a policy’s performance using logged contextual‑bandit data. It builds a PAC‑Bayes Bernstein lower‑confidence envelope over a threshold grid and all data‑dependent posteriors, returning an atom‑safe upper confidence bound on the selected posterior’s pooled p‑quantile. The authors provide finite‑sample guarantees, analyze sample complexity on a rare‑action class, and demonstrate the method’s effectiveness through simulations.
By Pauline Bourigault, Emmanuelle Bourigault
The paper introduces Normalizing Flow Maps (NFM), a new class of exactly invertible pushforward maps that extend normalizing flows and flow maps by learning the integrator of an underlying dynamical system while enabling efficient exact likelihood evaluation. NFMs incorporate two consistency axes—pointwise matching for β1-based regression losses and distributional matching for aligning pushforward distributions with a forward process—allowing scalable training. Experiments on CIFAR-10, ImageNet-64, and Boltzmann Generators demonstrate that NFMs achieve state‑of‑the‑art sample quality among parameter‑matched normalizing flows and outperform prior normalizing flows on peptide benchmarks.
By Francesco M. Ruscio, Timothy James Hitge, Fred Zhangzhi Peng, Danyal Rehman, Zander W. Blasingame, Nicholas M. Boffi, Michael M. Bronstein, Alexander Tong, Avishek Joey Bose
The paper introduces Lookahead Editing (LED), a method that enhances masked diffusion language models (MDMs) by adding a rewritable proposal state. LED performs multiple lookahead updates at each decoding step to refine future predictions before committing tokens, thereby improving expressivity while retaining MDMs’ simple transition schedule. Fine‑tuned LED models (LED‑7B and LED‑8B) achieve significant performance gains on eight benchmarks and outperform larger models such as LLaDA2.1‑16B using only supervised fine‑tuning.
By Yuyan Zhou, Zeyuan Yang, Yuxuan Song, Zhicheng Cai, Hongli Yu, Weiying Ma, Hao Zhou, James Kwok
CocoRNA is a cooperative multi-agent reinforcement learning framework that tackles RNA inverse design by breaking the problem into smaller sub‑problems solved collaboratively by multiple agents. A centralized critic provides global structural guidance during training, allowing the agents to jointly optimize their design strategies. Experiments on the Rfam dataset show that CocoRNA outperforms state‑of‑the‑art methods in both success rate and design speed, and additional tests on other biological sequence design tasks demonstrate its broad effectiveness.
By Tianmeng Hu, Biao Luo, Ke Li
The paper analyzes primal‑dual algorithms for constrained Markov decision processes under a two‑timescale framework, focusing on how non‑stationarity enters via additive and parametric channels. In the additive case, oscillating constraint levels cause the dual variable to saturate at a floor determined by the moving optimum, with amplification proportional to the timescale ratio. In the parametric case, oscillating constraint sensitivity leads to a parametric resonance at twice the natural frequency, driven by mismatches between primal and dual estimates; this resonance is mitigated by aligning the updates or using a PID term.
By Maxime Baelde
The paper investigates whether the policy gradient update used in many reinforcement learning algorithms—specifically the episode-summed, on‑policy update that omits the outer discount factor—can be interpreted as the gradient of any objective function. The authors prove that this update does not necessarily correspond to the gradient of any function and provide a counterexample where the update drives the policy toward a globally pessimal deterministic limit while the parameters diverge. They also highlight that this theoretical ambiguity has led to widespread misstatements in the literature, even in highly cited works from top conferences.
By Chris Nota, Philip S. Thomas
The paper introduces a hierarchical Bayesian framework for multi-task contextual bandits, capturing both heterogeneity and correlations among bandit instances. It proposes an empirical Bayesian method to estimate the covariance matrix of the prior, leading to two new algorithms—ebmTS and ebmUCB—that incorporate this prior into decision making. The authors provide frequentist regret bounds and show through synthetic and real-world experiments that these algorithms outperform existing methods, especially in complex settings.
By Xia Jiang, Rong J. B. Zhu
The paper introduces Cast‑R1, an 8‑billion‑parameter agentic forecasting model that learns to orchestrate analytical and prediction tools through trajectory distillation and multi‑turn reinforcement learning with structured memory. The training pipeline first generates multi‑turn interaction trajectories using a teacher LLM, then fine‑tunes the agent with supervised learning and group relative policy optimization, guided by rewards for accuracy and temporal structure. Experiments across several datasets show that Cast‑R1 effectively equips smaller language models with tool‑augmented, multi‑turn forecasting capabilities within structured workflows.
By Xiaoyu Tao, Mingyue Cheng, Yaguo Liu, Xingpeng Gao, Qingsong Wen, Shijin Wang, Enhong chen
arXiv:2609. 29694v2 Announce Type: replace Abstract: How much does high confidence cost when a multiclass learner sees only whether its prediction is correct?
By Guangjian Zhang, Zhengyi Yang, Xiaoyang Wang
LoongSage is a lightweight, fully asynchronous reinforcement learning framework designed for large language models that perform agentic tasks. It integrates with Megatron for distributed training and SGLang for high‑throughput inference, centralizing orchestration in a single controller and using Ray actors for GPU components. The framework introduces AgentFlow for unified execution of white‑box and black‑box agents, sliding‑window strategies to limit trajectory staleness, TransferMesh for efficient weight synchronization, and a memory‑efficient Teacher Manager that supports multi‑teacher on‑policy distillation. Experiments across four training runs show steady reward improvement, controlled train‑inference gaps, and sustained hardware utilization by eliminating synchronous idle periods.
By LoongSage Team, Bowei Song, Caiwei Zhao, Chuwen Liu, Dechuang Gu, Dong Fang, Feiyu Xie, Huaiyu Li, Hui Zhang, Jian Tian, Jian Wang, Jie Zhao, Liguang Li, Maoyun Wang, Meiyan Yang, Minghan Zhu, Pai Liu, Pandeng Yao, Renbao Tian, Ru Yang, Shuai Jia, Shuai Tan, Shuailong Li, Shuliang Li, Teng Jin, Tianze Yang, Wenbo Cui, Wenjing Bai, Wenkang Wu, Xiaowei Jiang, Xiaoyi Dong, Yanpeng Wang, Yanwen Wang, Yaohui Ma, Yawen Wang, Yihang Zhang, Yingjie Yan, Zhanbiao Lian, Zhenyu Lu, Zhida Hu, Zhihui Hao, Zhihui Hu, Zhiwen Xiao, Zhiyi Gao, Zhongwei Qi, Zhuangzhuang Wu, Ziqian Liu, Ziyue Zheng