arXiv AI October 19, 2026 By Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji

Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

Read the original on arXiv AI →

Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning proposes TreeOPSD, a critic‑free framework that treats each interaction turn as a tree node, branches alternative continuations, and recursively backs up terminal rewards to generate implicit turn‑level credit. The method maps this credit to a bounded teacher distribution based on the likelihood ratio to a frozen behavior policy, and minimizes the divergence between teacher and student to recover policy‑gradient directions while limiting updates across mini‑batches. Empirical results show TreeOPSD enhances agentic performance on long‑horizon benchmarks, notably improving Qwen3‑30B‑A3B‑Thinking‑2507’s accuracy by 4.97 points over GRPO in the Deep Research setting.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Sep 30

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.

By Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu, Fengran Mo, Jinghan Zhang, Mung Yao Jia, Gyuseok Lee, Yang Zhang, Na Wei, Dong Wang

llms reinforcement-learning efficiency benchmarks

arXiv AI

Aug 7

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

arXiv:2608. 05987v1 Announce Type: new Abstract: Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks.

By Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang

agents reinforcement-learning efficiency benchmarks

arXiv AI

Oct 15

LLM Value Functions for Credit Assignment in Agentic RL

arXiv:2610.15840v1 Announce Type: cross Abstract: Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assign...

By Kaijie Zhu, Weixiang Yan, Omar Attia, Abbas Kazerouni, Alexander Toshev, Andrew Szot

llms agents reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Sep 30

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...

By Hongyang Li, Xiao Li, Caesar Wu, Said Mammar, Gr\'egoire Danoy, Pascal Bouvry

llms reinforcement-learning safety

arXiv AI

Jul 31

ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks

arXiv:2606. 27814v4 Announce Type: replace Abstract: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.

By Qitai Tan, Zefang Zong, Mo Li, Yipeng Shi, Yang Li, Peng Chen

agents reinforcement-learning efficiency

Hugging Face Trending Papers

Jul 6

Weak-to-Strong Generalization via Direct On-Policy Distillation

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck.

reinforcement-learning efficiency

One email a morning, machine-written

Related stories

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

LLM Value Functions for Credit Assignment in Agentic RL

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks

Weak-to-Strong Generalization via Direct On-Policy Distillation