arXiv Machine Learning October 28, 2026 By Zhengping Jiang, William Jurayj, Liaoyaqi Wang, Chunsheng Zuo, Anqi Liu, Benjamin Van Durme

FOCUS: Active Critic Learning for Long-Horizon Language Model Agents

Read the original on arXiv Machine Learning →

FOCUS is a reinforcement learning method designed to improve long‑horizon language‑model agents by using selective branching to train a critic network more accurately. The rollout controller actively samples continuations from states that are expected to reduce the critic’s estimation error, and the refined value estimates are used to score segments between consecutive states with a TD(λ) advantage. Experiments on legal reasoning, deep search, and code localization tasks show that FOCUS outperforms baselines such as GRPO and VinePPO, achieving significant accuracy gains on benchmarks like BrowseComp‑Plus and DeonticBench.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Oct 15

LLM Value Functions for Credit Assignment in Agentic RL

arXiv:2610.15840v1 Announce Type: cross Abstract: Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assign...

By Kaijie Zhu, Weixiang Yan, Omar Attia, Abbas Kazerouni, Alexander Toshev, Andrew Szot

llms agents reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Sep 30

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

arXiv:2609.36178v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment o...

By Dongwon Jung, Hemanth Neelgund Ramesh, Yifan Wang, Xiaomin Li, Yuexing Hao, Yu Hu, Muhao Chen, Varun Chandrasekaran, Andrzej Banburski-Fahey, Jaron Lanier

llms agents reinforcement-learning

arXiv AI

Aug 21

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

arXiv:2608. 19842v1 Announce Type: new Abstract: Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models.

By Dayang Liang, Lang Feng, Bo An, Yunlong Liu

llms agents reinforcement-learning

arXiv AI

Sep 30

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...

By Hongyang Li, Xiao Li, Caesar Wu, Said Mammar, Gr\'egoire Danoy, Pascal Bouvry

llms reinforcement-learning safety

Hugging Face Trending Papers

Aug 20

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks.

llms agents reinforcement-learning

arXiv AI

Jun 10

3SPO: State-Score-Supervised Policy Optimization for LLM Agents

arXiv:2606. 09961v1 Announce Type: cross Abstract: Training large language models (LLMs) as autonomous agents via reinforcement learning (RL) has enabled frontier models to achieve superhuman performance in long-horizon tasks.

By Yu Han, Kailing Li, Yang Jiao, Yulin Dai, Yuqian Fu, Linhai Zhuo, Tianwen Qian

llms agents reinforcement-learning

One email a morning, machine-written

Related stories

LLM Value Functions for Credit Assignment in Agentic RL

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

3SPO: State-Score-Supervised Policy Optimization for LLM Agents