FOCUS is a reinforcement learning method designed to improve long‑horizon language‑model agents by using selective branching to train a critic network more accurately. The rollout controller actively samples continuations from states that are expected to reduce the critic’s estimation error, and the refined value estimates are used to score segments between consecutive states with a TD(λ) advantage. Experiments on legal reasoning, deep search, and code localization tasks show that FOCUS outperforms baselines such as GRPO and VinePPO, achieving significant accuracy gains on benchmarks like BrowseComp‑Plus and DeonticBench.
arXiv:2610.15840v1 Announce Type: cross Abstract: Reinforcement learning (RL) for large language model (LLM) agents remains challenging in long-horizon tasks with sparse outcome rewards, where assign...
arXiv:2609.36178v1 Announce Type: cross Abstract: Group Relative Policy Optimization (GRPO) has become a promising approach for training large language model agents. However, its uniform assignment o...
arXiv:2608. 19842v1 Announce Type: new Abstract: Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models.
arXiv:2609.37119v1 Announce Type: cross Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...
Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks.
arXiv:2606. 09961v1 Announce Type: cross Abstract: Training large language models (LLMs) as autonomous agents via reinforcement learning (RL) has enabled frontier models to achieve superhuman performance in long-horizon tasks.
The paper introduces POISE, a reinforcement learning algorithm that uses a model’s internal states as a value estimator to reduce variance in reinforcement learning with verifiable rewards (RLVR). By employing a lightweight probe that reads internal signals during the forward pass, POISE predicts baselines online and uses a cross‑rollout construction to keep gradients unbiased. Experiments on Qwen3‑4B and OLMo3‑7B‑Instruct‑DPO across six domains show POISE outperforms existing RLVR baselines, offering more stable training and a value model that generalizes across tasks and scales with the policy.
arXiv:2610.19103v1 Announce Type: new Abstract: Group-based reinforcement learning enables large language model agents to learn from sparse outcome rewards in long-horizon tasks. Recent methods deriv...
arXiv:2606. 25451v1 Announce Type: new Abstract: Estimating token-level advantages in reinforcement learning (RL) for language models remains challenging because scaling up episodic experience collection is expensive.
The paper introduces Actor‑Critic with Action Chunking (AC2), a method that assigns credit to short action chunks instead of entire trajectories, enabling policy updates without waiting for terminal rewards. AC2 employs local readiness, reference solutions, and 10k‑token chunks to make critic‑based credit assignment reliable. Experiments on Qwen3‑4B with FineProofs‑RL show AC2 surpasses GRPO’s peak validation score while using 2.5× fewer decoding FLOPs and fewer training steps.
Group-Graph Policy Optimization (G2PO) is a novel group-based reinforcement learning algorithm designed for long-horizon agentic tasks. It transforms linear interaction trajectories into a global state‑transition graph, aggregates identical observations across trajectories to reduce sampling variance, and redefines actions as transitions between state nodes. By employing an edge‑centric advantage estimation and globally standardizing TD errors, G2PO identifies and prioritizes critical transitions, achieving up to 22.2% higher success rates than state‑of‑the‑art baselines on benchmarks such as WebShop, ALFWorld, and AppWorld.
arXiv:2609.37825v1 Announce Type: cross Abstract: Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse...