arXiv Machine Learning

ProCredit: From Outcome Rewards to Progress Credit in Agentic Reinforcement Learning

arXiv Machine Learning
Oct 19

Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents

The paper introduces Grounded Credit Assignment (GCA) for reinforcement learning in multi‑turn search agents. GCA separates credit for evidence retrieval and answer correctness, correcting the bias in GRPO that favors evidence‑retrieving rollouts only proportionally to accuracy gains. Applying GCA to Search‑R1 boosts average EM by 5.7–6.3 points and increases the proportion of evidence‑supported correct answers from 76.9% to 93.4%.

By Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang, Weihong Luo, Yuyu Luo
arXiv Machine Learning
Aug 31

VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning

The paper introduces VICT, a method that leverages the internal structure of verifiable tasks to perform fine‑grained credit assignment for long‑horizon LLM agents. VICT exposes executable or evidence‑backed atoms from a task’s terminal verifier and traces them back to actions via dependency‑valid proof edges, redistributing advantage only along these edges. This approach improves performance on ALFWorld and WebShop compared to outcome‑only training and matches recent fine‑grained credit methods without requiring additional critics, labels, or inference‑time verifier access.

By Pengcheng Li, Zhengyang Zhang, Dongxu Zhang, Sui Huang, Shaohua Ma
arXiv AI
Nov 6

AttnSR: Attention Step Reward for Long-Horizon Agentic RL

AttnSR introduces a step-level reward augmentation technique for long-horizon language model agents, deriving reward allocations from policy attention during training replay without extra reward models or rollouts. By tracing backward through the policy attention graph, it estimates the relevance of earlier steps to the final outcome, using the terminal step relative to a baseline to reinforce or penalize selected steps. Experiments on Qwen2.5-7B-Instruct show AttnSR consistently outperforms PPO on ALFWorld, raising held‑out success from 91.2% to 97.1%, and it also improves ScienceWorld while reducing environment interactions by 29.9%.

By Mengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou
arXiv Machine Learning
Jul 16

TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

arXiv:2607. 13988v1 Announce Type: new Abstract: Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training.

By Leitian Tao, Baolin Peng, Wenlin Yao, Tao Ge, Hao Cheng, Mike Hang Wang, Jianfeng Gao, Sharon Li