← Back to all news
arXiv Computation and Language By Jixuan Leng, Si Si, Fei Wang, Cho-Jui Hsieh, Jiong Zhang, Hsiang-Fu Yu, Inderjit S. Dhillon

CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

Read the original on arXiv Computation and Language →

The paper introduces CONCUR, an advantage estimator designed for multi‑task, multi‑reward reinforcement learning with large language models. CONCUR replaces standardized scores of gated auxiliary rewards with adjusted scores that ensure higher primary rewards never receive lower scores and that responses at the primary reward ceiling receive equal scores. Two variants, CONCUR‑T and CONCUR‑A, are evaluated on models across math, code, instruction following, and writing tasks, outperforming all baselines and the approach of dropping all gated auxiliaries.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 30

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.

By Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu, Fengran Mo, Jinghan Zhang, Mung Yao Jia, Gyuseok Lee, Yang Zhang, Na Wei, Dong Wang
llmsreinforcement-learningefficiencybenchmarks
More like this →
arXiv Computation and Language
Oct 27

Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization

arXiv:2610.34862v1 Announce Type: cross Abstract: LLM post-training often optimizes multiple objectives with unequal priorities, such as correctness and reasoning efficiency. However, the widely used...

By Xiangchi Yuan, Xingjian Diao, Zheyuan Liu, Chunhui Zhang, Dachuan Shi, Kejing Xia, Zhenbang Du, Shenglong Yao, Soroush Vosoughi, Wenke Lee
llmsreinforcement-learning
More like this →
arXiv AI
Oct 16

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

The paper investigates how combining multiple rewards during large‑language‑model post‑training can lead to a failure called multi‑reward hacking, where the overall reward improves while the primary task performance worsens. The authors show that symmetric reward aggregation can cause the policy to favor auxiliary rewards that are more reliably obtained, regardless of initialization. They propose Trust‑Anchored Gating (TAG), which selects an anchor reward channel and gates auxiliary credit based on anchor satisfaction, thereby preventing auxiliary‑only outcomes and restoring primary‑task performance. Experiments on Qwen and Llama models demonstrate that TAG significantly mitigates degradation seen with additive, constrained, or normalized aggregation, achieving higher pass rates and shorter responses compared to baseline methods.

By Yushu Li, Jiin Woo, Xiaoxiao Li, Youngsuk Park
llms
More like this →