The paper introduces ReTRAIL, a framework that allows long‑horizon agents to rewind the environment to earlier states while preserving forward progress in knowledge. By training a shared policy with State‑Anchored GRPO and a branch‑advancement reward, ReTRAIL enables agents to recover from unproductive states and reuse insights from abandoned exploration. Experiments on MineSweeper, Sokoban, ALFWorld, and ScienceWorld show that this combined rewinding and experience‑reuse strategy outperforms simple rewind primitives and improves long‑horizon task solving.
By Ziang Ye, Wentao Shi, Lei Chen, Zhengzhou Cai, Yu Wang, Zhiyuan Yao, Zihan Wang, Xierui Song, Fuli Feng, Yixin Cao
arXiv:2607. 21419v1 Announce Type: new Abstract: In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization.
By Yipeng Shi, Zhipeng Ma, Yue Wang, Qitai Tan, Yang Li, Peng Chen, Zhengzhou Zhu
arXiv:2512. 07287v3 Announce Type: replace-cross Abstract: As intents unfold and environments change, multi-turn agents face continuously shifting decision contexts.
By Sijia Li, Yuchen Huang, Zifan Liu, Zijian Li, Jingjing fu, Lei Song, Jiang Bian, Jun Zhang, Rui Wang
The paper introduces Agent Evolving Learning (AEL), a two‑timescale framework that dynamically evolves an LLM agent’s memory‑retrieval harness in open‑ended environments. A fast Thompson‑Sampling bandit selects among retrieval policies each episode, while a slower LLM reflection diagnoses performance drops and injects new policies when the current set plateaus. AEL outperforms ten self‑improving and non‑LLM baselines on a sequential portfolio benchmark, boosting Sharpe ratio by 27% and achieving significant accuracy gains on a support‑ticket routing stream.
By Wujiang Xu, Jiaojiao Han, Minghao Guo, Kai Mei, Xi Zhu, Han Zhang, Dimitris N. Metaxas
arXiv:2610.27495v1 Announce Type: cross
Abstract: Self-evolving agents aim to improve from their own experience rather than remain fixed after deployment, but naturally collected trajectories often l...
By Yuhang Zhou, Yifan Wu, Bo Peng, Mingyi Wang, Jiayi Liu, Xiangjun Fan, Lizhu Zhang, Zhuokai Zhao
arXiv:2510.15047v2 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) as agents often fail to improve in new environments. We identify and characterize a failure mode we call explora...
By Shiqi Chen, Tongyao Zhu, Zian Wang, Jinghan Zhang, Kangrui Wang, Ruochen Zhou, Siyang Gao, Teng Xiao, Yee Whye Teh, Junxian He, Manling Li
The paper introduces Online Meta-Control of Experience (OMCE), a framework that updates an agent’s experience memory and learns a meta-control policy during test-time adaptation. OMCE uses a lightweight contextual-bandit meta-controller to choose among four cognitive actions—DIRECT, EPISODE, GUIDE, and DISTILL—based on compact retrieval signals, thereby deciding when to invoke experience, which memory format to use, and when to consolidate new guidance. Experiments on four online adaptation benchmarks across four backbones show that OMCE can improve first-attempt success by up to 20.5 percentage points over the strongest baseline while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT‑5.2, and controlled ablations confirm the benefits of learned routing and ongoing policy adaptation.
By Zeyuan Zhao, Yuanyuan Shi, Peng Yu, Lingyue Fu, Shao Zhang, Ying Wen
Headroom-Drift Replay is a replay control primitive designed for GRPO that separates reuse into two decisions: Headroom ranks stored groups by remaining learning value, while Drift gates them by compatibility with the current policy. The method keeps the fresh on‑policy stream unchanged and adds no auxiliary generation or training machinery. Across mathematical reasoning, multimodal reasoning, and Agentic Search benchmarks, it outperforms naive replay and matches or exceeds broader replay methods on Avg Mean@32, delivering comparable quality at materially lower wall‑clock time in Agentic Search.
By Hyun Bin Park, Du-Seong Chang
arXiv:2610.27012v1 Announce Type: cross
Abstract: Reinforcement learning with verifiable rewards (RLVR) trains a language model by sampling responses, scoring them with an automatic verifier, and upd...
By Yunan Zeng, Jinhao Jing, Xi Cheng, Yuwang Wang
TIGPO (Temporal Instance-Graph Policy Optimization) extends graph-based credit assignment for long-horizon LLM agents by maintaining a persistent transition graph per task across policy updates. It allocates rollout budgets to both new exploration and revisiting past tasks, pairing current rollouts with earlier ones to create cross‑temporal references that stabilize advantage estimation. Experiments on ALFWorld and WebShop show TIGPO consistently outperforms previous group‑based and graph‑based policy optimization methods.
By Jinwei Gan
arXiv:2608.23318v1 Announce Type: new
Abstract: Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rol...
By Zixuan Wang, Yanrui Miao, Zhengxi Lu, Teng Pan, Yiwen Qiu, Hongxing Li, Peng Qiu, Ruiqing Zhang, Yongliang Shen
The paper introduces Self‑Retrospection Distillation (SRD), a method that uses post‑hoc trajectories to supervise foresight predictions made before an agent acts. SRD distills information from completed rollouts into trajectory‑blind predictions, enabling the agent to anticipate useful actions and avoid pitfalls without requiring explicit foresight at inference time. Experiments on ten tool‑integrated reasoning and long‑horizon tasks show that SRD improves performance by up to 24.2 percentage points, especially when reward signals are scarce or uniform across rollouts.
By Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu