← Back to all news
arXiv Machine Learning By Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman, Samuel Okafor

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

Read the original on arXiv Machine Learning →

The paper introduces TWICE (TWIn-world Credit Estimation), a method for agentic reinforcement learning that uses twin-world counterfactuals to assign credit to individual decisions within an episode. By re‑running episodes with the same sampling noise but different decisions, TWICE produces unbiased turn‑level advantages for policy‑gradient updates, reducing the number of rollouts needed compared to independent counterfactuals. Experiments on seven question‑answering benchmarks show that TWICE improves exact match scores for GRPO by 5.3 and 4.5 points on Qwen2.5‑3B and 7B, outperforms tree‑search and counterfactual‑branching baselines, and requires 68% fewer auxiliary tokens than VinePPO.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 28

Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning

The paper investigates the problem of sharing a single critic across multiple parallel environments in reinforcement learning. It shows that when environments assign different expected returns to the same state, a shared critic must reconcile conflicting value targets, which can distort advantage estimates and misguide policy updates. The authors propose a simple fix—providing the critic with the environment index—demonstrating through bandit models and experiments on CartPole, MuJoCo, BipedalWalker, and 16 Procgen games that this conditional critic stabilizes learning and boosts returns, achieving a 40.8% improvement in aggregate normalized return on unseen levels.

By Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen
reinforcement-learning
More like this →
arXiv AI
Oct 5

$T^5$: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training

arXiv:2609.32791v2 Announce Type: replace Abstract: Reinforcement mid-training lets language models learn internal thoughts from unlabeled text, but efficient token-level credit assignment remains ch...

By Nan Qiao, Yebin Yang, Weinong Wang, Shuning Wang, Shangpin Peng, Fengyuan Lu, Xinming Wang, Zhehan Kan, Ruixu Zhang, Songyang Zhang, Sheng Yue, Yonglong Tian, Ju Ren
llmsreinforcement-learningbenchmarks
More like this →