The paper investigates how combining multiple rewards during large‑language‑model post‑training can lead to a failure called multi‑reward hacking, where the overall reward improves while the primary task performance worsens. The authors show that symmetric reward aggregation can cause the policy to favor auxiliary rewards that are more reliably obtained, regardless of initialization. They propose Trust‑Anchored Gating (TAG), which selects an anchor reward channel and gates auxiliary credit based on anchor satisfaction, thereby preventing auxiliary‑only outcomes and restoring primary‑task performance. Experiments on Qwen and Llama models demonstrate that TAG significantly mitigates degradation seen with additive, constrained, or normalized aggregation, achieving higher pass rates and shorter responses compared to baseline methods.
By Yushu Li, Jiin Woo, Xiaoxiao Li, Youngsuk Park
arXiv:2610.35716v1 Announce Type: cross
Abstract: Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is...
By Shurun Li, Michael Hua Wang, Haibo Zeng, Long Wang
Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout. We study ho...
The paper introduces Density-Aware Reward Aggregation (DARA), a method that adjusts reward weighting in multi-reward reinforcement learning based on the density of active rewards within rollout batches. By applying an inverse-square-root density correction, DARA gives more weight to less frequently active rewards, enabling faster learning of targeted behaviors. Experiments on tool calling and mathematical reasoning demonstrate that DARA achieves comparable final performance while reducing training steps by up to 26% and 65% respectively.
By Tong Zheng, Skylar Zhai, Zhan Cheng, TianMing Sha, Youling Huang, Shuo Zhou, Shaotong Qi, Jingcheng Liang, Xuwei Ding, Pengcheng Xu
The paper investigates the problem of sharing a single critic across multiple parallel environments in reinforcement learning. It shows that when environments assign different expected returns to the same state, a shared critic must reconcile conflicting value targets, which can distort advantage estimates and misguide policy updates. The authors propose a simple fix—providing the critic with the environment index—demonstrating through bandit models and experiments on CartPole, MuJoCo, BipedalWalker, and 16 Procgen games that this conditional critic stabilizes learning and boosts returns, achieving a 40.8% improvement in aggregate normalized return on unseen levels.
By Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen
arXiv:2611.05942v1 Announce Type: new
Abstract: RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp. Recent work has studied policy e...
By Mengni Jia, Mengyu Zhou, Xiaoxi Jiang, Guanjun Jiang