arXiv AI
Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 6
ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment
arXiv:2608. 05102v1 Announce Type: new Abstract: Long-horizon search agents must make multiple sequential actions (steps) to search, retrieve, verify, and integrate evidence to reach a final answer.
arXiv AI
Aug 14
Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
arXiv:2608. 12764v1 Announce Type: cross Abstract: Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment.
arXiv Computation and Language
Oct 27
SearchJev: A Calibrated System-1 Model for Search Agents
arXiv:2610.05107v2 Announce Type: replace-cross Abstract: Search agents repeatedly make short decisions about relevance, evidence sufficiency, and search actions. Using generative language models for...
arXiv Computation and Language
Sep 2
AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
AdaSearch introduces a two‑stage reinforcement learning framework that separates problem solving from the decision to search in large language models. By using an F1‑based decision metric, it explicitly evaluates when external search is needed, reducing unnecessary search calls while maintaining high question‑answering performance. Experiments show that AdaSearch improves search‑decision quality with only a minor impact on accuracy compared to always‑search strategies.
Hugging Face Trending Papers
Sep 3
Spurious Advantage Hidden in GRPO
The paper identifies a flaw in Group Relative Policy Optimization (GRPO) called the spurious advantage, where the advantage estimator can incorrectly reward rollouts that reach correct answers by guessing rather than reasoning. This issue appears in bounded-answer tasks, open-answer sets with bounded sub-cases, and search agents with many paths to the same answer. The authors propose SIGNBALANCE, a new magnitude estimator that removes composition, maintains verifier sign, applies a global scale, and uses stop‑gradient per‑class rescaling to restore zero‑mean balance, achieving comparable or better performance on math and search benchmarks.
arXiv AI
Jun 8
SlimSearcher: Training Efficiency-Aware Web Agents via Adaptive Reward Gating
arXiv:2606. 07074v1 Announce Type: cross Abstract: Deep research agents have demonstrated remarkable capabilities in complex information-seeking tasks, yet this power comes at a steep computational cost.