arXiv Machine Learning November 2, 2026 By Rui Yu, Chong Mu, Jinchuan Zhang

TSBPO: Tree-Structured Branching for Policy Optimization in Interactive Language Agents

Read the original on arXiv Machine Learning →

The paper introduces TSBPO, a tree‑structured sampling‑to‑update framework for training interactive language agents. It tackles three intertwined challenges—sampling task and state continuations, assigning delayed feedback, and aggregating token losses—by using a likelihood‑fitted Root/Branch allocator, trajectory and observation‑group advantages, and a clipped token surrogate aggregation. Experiments with a Qwen2.5‑1.5B actor show high success rates on ALFWorld and WebShop benchmarks, demonstrating the effectiveness of the proposed training configuration.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 10

Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents

arXiv:2608. 06861v1 Announce Type: new Abstract: Training large language model agents in long-horizon environments requires assigning credit from sparse terminal outcomes to individual actions.

By Hongxi Yan, Ziyue Huang, Shichao Fan, Qingjie Liu

llms agents multimodal

arXiv AI

Jun 10

TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning

arXiv:2606. 11119v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.

By Heming Zou, Qi Wang, Yun Qu, Yuhang Jiang, Lizhou Cai, Yixiu Mao, Ru Peng, Xin Xu, Weijie Liu, Kai Yang, Saiyong Yang, Xiangyang Ji

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 15

DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter

arXiv:2610.17666v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps....

By Xuyan Ye, Xiyan Fu, Shengda Fan, Jiawei Cui, Yankai Lin, Wei Lu

llms agents reinforcement-learning

arXiv Computation and Language

Oct 15

TreeFlex: Agents Navigating Their Own Search Trees

arXiv:2610.20334v1 Announce Type: cross Abstract: A ReAct-style agent accumulates observations, intermediate results, and failed attempts in a single context. As this context grows, stale and off-pat...

By Dongzhuo Li, Fangzhou Cheng, Weiwei Li, Kevin Han, Haixu Ma, Zhongchang Sun, Ruiyi Li, Serena Li, Xiangjun Fan, Lizhu Zhang, Yuandong Tian

llms agents reinforcement-learning benchmarks

arXiv AI

Aug 26

SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

SPO++ is a new reinforcement learning method that eliminates the need for waiting on sibling rollouts by using a persistent prompt-level value estimate. It corrects a mismatch in the original SPO approach by standardizing terminal-outcome advantages under an action-token measure and reorganizes prompt evidence based on the policy event that generated it. Experiments on ALFWorld and Math‑TIR show that SPO++ improves online learning efficiency, with action-token-measure normalization identified as the key component.

By Kai Ruan, Jinghao Lin, Qianshan Wei, Ziqi Zhou, Zihe Huang

agents reinforcement-learning benchmarks

arXiv AI

Oct 15

PTGPO: Prefix Tree Group Policy Optimization

arXiv:2610.14737v1 Announce Type: cross Abstract: Training language agents on long-horizon interactive tasks is challenging because sparse terminal rewards provide weak credit signals for intermediat...

By Ruolin Chen, Yang Li, Jihang Wang, Zhuoya Zhao, Yuchen Liu, Jiaming Xu, Qian Zhang

agents reinforcement-learning benchmarks

One email a morning, machine-written

Related stories

Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents

TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning

DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter

TreeFlex: Agents Navigating Their Own Search Trees

SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

PTGPO: Prefix Tree Group Policy Optimization