The paper introduces TACE, a two‑stage estimator for advantage estimation in Group Relative Policy Optimization. Stage S1 replaces linear standardization with an optimal‑transport map to a standard normal, aligning reward distributions across tasks and limiting outliers. Stage S2 rank‑calibrates token‑level credit assignment, normalizing token weights to preserve total credit while redistributing it within trajectories. The authors prove theoretical guarantees—inter‑task moment equity, deterministic tail bounds, and mass‑conservation—that previous methods lack, and demonstrate that TACE achieves the best performance on most benchmarks in a multimodal reasoning suite.
arXiv:2606. 00257v1 Announce Type: cross Abstract: Token-level credit assignment for language-model reinforcement learning is usually formulated as if the policy were fully trainable, while practical LLM-RL pipelines often rely on parameter-efficient fine-tuning, especially LoRA.
arXiv:2610.18704v1 Announce Type: new Abstract: In this paper, we propose an extremely efficient, training-free method to extract token-level reward signals directly from an existing deep reward mode...
arXiv:2601. 07408v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has emerged as a promising critic-free reinforcement learning paradigm for reasoning tasks.
The paper introduces Outcome-Conditioned Visual Credit (OVC), a plug‑in framework for multimodal reinforcement learning that separates visual association measurement from token‑level credit assignment. OVC uses Relative Visual Encoding to create a context‑relative visual coordinate and Outcome‑Conditioned Evidence Calibration to adjust token contributions based on outcome‑discriminative evidence. When applied to GSPO-token, OVC improves mean accuracy by 1.86–2.39 percentage points across six benchmarks without requiring additional reward models or inference‑time modules.
arXiv:2606. 25451v1 Announce Type: new Abstract: Estimating token-level advantages in reinforcement learning (RL) for language models remains challenging because scaling up episodic experience collection is expensive.
arXiv:2611.05942v1 Announce Type: new Abstract: RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp. Recent work has studied policy e...
arXiv:2609.39168v1 Announce Type: new Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet existing...
The paper introduces Selective Successor Eligibility Traces (S-trace), a critic‑free eligibility trace method designed to improve temporal credit assignment in Reinforcement Learning with Verifiable Rewards (RLVR). S-trace reweights successor importance to recover downstream credit signals under PPO-style clipping and selectively propagates them to historical tokens using entropy-based token selection. Experiments on Qwen3 models show that S-trace yields significant absolute gains in average pass@16 over GRPO and trace-style GRPO, while also producing shorter responses.
arXiv:2606. 18810v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps.
arXiv:2609.40360v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has improved the reasoning capabilities of large language models (LLMs), yet their predictions...
The paper introduces Stepwise Marginal Information Gain (MIG), an intrinsic process reward that evaluates how each reasoning step of a large language model (LLM) or vision-language model (VLM) improves the likelihood of the reference answer. MIG rewards only new likelihood maxima, preventing duplicate credit, and is combined with outcome, format, and self‑distillation objectives to guide training. Experiments on eight benchmarks show that this method outperforms outcome‑only reinforcement learning and improves accuracy by up to 4.8 points over binary‑reward training, including a 12.6‑point gain on MathVerse and a 12.9‑point advantage on vision‑language transfer at 7B parameters.
The paper introduces CrEST, a hierarchical credit assignment framework for multi-turn, multi-step large language model agents. CrEST anchors each token’s advantage sign to verifier-defined rewards while using a self-teacher to provide dense magnitude signals, addressing both inter-turn dilution and intra-turn token contribution issues. Experiments on BFCL V3 and WildToolBench show CrEST consistently outperforms reinforcement learning and distillation baselines, especially on long trajectories and strict session-level metrics.