arXiv AI By Dongwon Jung, Hemanth Neelgund Ramesh, Yifan Wang, Xiaomin Li, Yuexing Hao, Yu Hu, Muhao Chen, Varun Chandrasekaran, Andrzej Banburski-Fahey, Jaron Lanier

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv AI
Aug 26

Contrastive Branch Policy Optimization

Contrastive Branch Policy Optimization (CBPO) is a reinforcement learning method that separates the allocation of a fixed rollout budget from the translation of branch outcomes into token-level credit. It uses generation entropy to screen branch positions, path- and node-level decay to distribute the budget, and Contrastive Branch Value (CBV) to estimate local decision sensitivity without changing reward signs. CBPO partitions trajectories into non-overlapping credit segments, preventing duplicated gradients and enabling fine-grained credit assignment using only outcome rewards.

By Ying Wang, Changlin Qiu, Bang Lin, Linbo Jin, Wen Jiang, Zhe Sun, Jingli Yang
arXiv AI
Sep 1

Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization

The paper introduces TASPO, a method that transforms privileged information (PI) into outcome‑grounded action credit for language‑model agents. TASPO constructs decision‑applicable PI from verified successful experience, aggregates PI‑induced likelihood shifts at the executable‑action level, and converts relative action support into positive, bounded, mean‑preserving weights on the original trajectory advantage. Experiments on three agentic benchmarks show TASPO improves over GRPO by 10.6% and generalizes better to unseen tasks, while reducing supervision mismatch and stabilizing policy optimization.

By Jingxiao Yang, Wangjie Gan, Yingxuan Zhuang, Wenqi Zhang, Jintao Chen, Xuhong Zhang
arXiv Machine Learning
Oct 22

One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents

The paper introduces Effect-Marginal Policy Optimization (EMPO), a method that addresses credit-update misalignment in multi-turn reinforcement learning for large language model agents. EMPO aligns policy updates with execution-level equivalence by using an effect-level surrogate that couples updates across different textual responses that produce the same effect. Experiments on ALFWorld, WebShop, and Sokoban demonstrate that EMPO improves both task performance and learning efficiency compared to strong baselines.

By Shaojin Ma, Ruitao Wang, Pengyi Li, Jie Yang, Liangyu Chen, Gangyi Zhao, Hongyao Tang, Yan Zheng, Jianye Hao