The paper introduces Distill Until Divergence, a method for agentic reinforcement learning that uses error-truncated hindsight distillation. It routes two complementary hindsight skills—Positive for successful workflows and Negative for failure corrections—based on trajectory outcomes and error boundaries. The post-error segment is optimized solely by GRPO, preserving outcome-based learning while avoiding mismatched hindsight supervision.
arXiv:2608. 07371v1 Announce Type: new Abstract: Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards.
arXiv:2610.07349v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) avoids a separate critic by estimating advantages from rollout groups. For multi-turn agents, however, traj...
Prospective Hindsight (PH) is a self‑calibrating training principle for reinforcement learning that augments any retrospective base method with a signal derived from the gap between an agent’s prospective prediction and its retrospective evaluation. By weighting gradients with a stop‑gradient surprise based on this prediction‑reality gap, PH highlights samples where the agent’s self‑model is most inaccurate and encourages the policy and predictor to co‑evolve, progressively focusing on remaining blind spots. Experiments on single‑turn verifiable tasks and a multi‑turn personal‑agent task show that PH improves both task performance and calibration across model scales, addressing overconfident failures in single‑turn settings and underconfident successes in multi‑turn settings.
arXiv:2608. 01597v1 Announce Type: new Abstract: Search-augmented LM agents are typically trained with a binary exact-match reward, which throws away most of what a failed trajectory tells us about why it failed.
AHEAD is a step‑aware framework that augments reinforcement learning for multi‑turn LLM agents by matching different supervision sources to different step types. The teacher receives environment feedback on all steps and LLM‑generated corrective hints only on error steps, providing finer‑grained guidance than uniform trajectory‑level rewards. Across ALFWorld, WebShop, and Search‑based QA, AHEAD improves task success by 13.3 points on ALFWorld and 11.0 on WebShop at 7B, reaches target success rates faster, and solves tasks within tighter interaction budgets compared to outcome‑only RL and prior self‑distillation baselines.
TreeHCA introduces a method that couples hindsight-guided exploration with hindsight credit assignment for multi-turn large language model agents. By using changes in reference-answer likelihood to revisit intermediate histories and sample alternative continuations, it builds a tree of reasoning and tool interactions, and applies an inverse-hindsight importance correction to propagate terminal outcomes into turn-level advantages. Experiments on seven question-answering benchmarks with three Qwen backbones show that TreeHCA outperforms all baselines in aggregate exact match.
HINT-SD introduces a targeted self‑distillation framework for long‑horizon language‑model agents that uses full‑trajectory hindsight to identify failure‑relevant actions and applies feedback‑conditioned distillation only to those action spans. This selective approach reduces the need for per‑turn feedback, improving training efficiency and effectiveness. Experiments on BFCL v3 and AppWorld demonstrate that HINT‑SD outperforms dense per‑turn feedback baselines by up to 13.60 percentage points on average while cutting training time per step by 2.26×.
arXiv:2608. 05987v1 Announce Type: new Abstract: Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks.
arXiv:2607. 28076v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents.
SCAD (Structured Credit Assignment and Distillation) is a new method for training long‑horizon agents that separates planning from bounded subtask execution, distills execution locally, and refines planning credit using cross‑rollout subtask prefix trees. It gives planning full terminal credit while providing execution with positive terminal credit and teacher guidance. On all tested benchmarks, SCAD improves macro‑average accuracy by 4.48 percentage points on text tasks and 4.19 points on multimodal tasks compared to the strongest baseline.
arXiv:2606. 18810v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps.
arXiv:2606. 16285v1 Announce Type: cross Abstract: Long-horizon agents rely on memory mechanisms to compress interaction history, but optimizing memory writing faces a distinct credit assignment challenge: a memory update may be rewarded or penalized due to downstream tool failures, noisy observations, or reasoning errors rather than its own contribution.