CORD is a conflict‑aware optimization framework that addresses gradient cancellation between task rewards and teacher guidance in joint reinforcement learning and self‑distillation. It separates the protected objective from the projection trigger, applies a nearest‑point closed‑form update per mini‑batch, and offers gradient diagnostics to identify and correct conflicts. Experiments across three training recipes show that CORD improves ALFWorld success by 1.5–15.9 percentage points and raises WebShop SDAR success from 60.7% to 64.6%.
arXiv:2609.39306v1 Announce Type: cross Abstract: Iterative self-distillation enables LLM agents to learn from successive deployments, offering a path toward recursive self-improvement (RSI). Yet our...
Iterative self-distillation enables LLM agents to learn from successive deployments, offering a path toward recursive self-improvement (RSI). Yet our experiments with existing methods reveal a collaps...
AHEAD is a step‑aware framework that augments reinforcement learning for multi‑turn LLM agents by matching different supervision sources to different step types. The teacher receives environment feedback on all steps and LLM‑generated corrective hints only on error steps, providing finer‑grained guidance than uniform trajectory‑level rewards. Across ALFWorld, WebShop, and Search‑based QA, AHEAD improves task success by 13.3 points on ALFWorld and 11.0 on WebShop at 7B, reaches target success rates faster, and solves tasks within tighter interaction budgets compared to outcome‑only RL and prior self‑distillation baselines.
arXiv:2610.19789v1 Announce Type: cross Abstract: Self-distillation in agentic reinforcement learning has produced a growing number of methods. However, existing evaluations often reuse published bas...
arXiv:2608. 05219v1 Announce Type: new Abstract: Privileged on-policy distillation provides dense supervision for multi-turn agents by allowing a synchronized teacher to re-score the student's response at every turn with access to training-only references, such as successful trajectories.
The paper investigates on‑policy distillation (OPD) as a preparatory step for reinforcement learning (RL). It shows that students initialized with OPD achieve higher final RL performance than those trained directly with RL or with supervised fine‑tuning followed by RL, even when OPD offers little immediate accuracy gain. The study also finds that the choice of distillation objective (reverse‑KL vs forward‑KL) and the source of trajectories influence OPD’s effectiveness at different stages of RL training.
arXiv:2608. 12957v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect.
The paper introduces Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation (Road MOPD), a method that adds non-routed teachers as auxiliary cross-domain critics to address gradient conflicts in multi-teacher distillation. It employs token-level credit assignment based on teacher agreement and preventive gradient projection to coordinate updates and reduce conflicts. Experiments show that Road MOPD improves performance, especially in high-conflict scenarios, and mitigates degradation in disadvantaged domains.
arXiv:2606. 14368v1 Announce Type: new Abstract: We study multi-domain LLM training in which two models, each stronger in a different domain, co-evolve by tutoring each other through on-policy feedback.
arXiv:2608. 07371v1 Announce Type: new Abstract: Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards.
arXiv:2610.00838v1 Announce Type: cross Abstract: Agentic reinforcement learning (RL) trains a large language model (LLM) to act over long, multi-step interactions. However, a single localized error...
arXiv:2606. 27814v4 Announce Type: replace Abstract: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.