arXiv AI

CAFE: Self-Improving Search Agents Need Co-Evolving Feedback

CAFE (Coupled Agent–Feedback Evolution) is a framework that lets a shared‑parameter model alternate between acting as a search agent and as a critic that provides corrective feedback. By learning when to request feedback and how to use it, CAFE trains the agent to recover from its own failures and shapes rewards both online and offline. Experiments on seven search benchmarks show that CAFE outperforms other RL‑based agents, maintains gains on out‑of‑domain tests, and reduces hallucinations, indicating that co‑evolving feedback is essential for self‑improving search agents.

arXiv AI
Oct 15

Search-G1: Co-Evolving Representation-Based Rewards for Selective Grounded Search

Search-G1 is a reward framework that encourages language agents to retrieve external information only when necessary and to ground their answers in that evidence. It uses two hidden‑state readouts—one predicting whether a closed‑book answer suffices and another measuring sensitivity to evidence deletion—to calibrate retrieval necessity and evidence reliance. The reward system gates evidence credit by necessity, rewards sufficient direct answers, and penalizes redundant searches, all without requiring costly annotations or LLM‑judge inference.

By Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang, Junming Zhang, Ranjie Duan, Qiaolin Xia, Hao Wang, Yu Lu, Haibo Shi, Xingjun Ma
arXiv Computation and Language
Oct 12

Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting

The paper investigates reflective prompt optimization (GEPA) for a multi‑agent deep‑research language‑model, focusing on a topic‑disjoint train/validation/test split to avoid memorization. Across three random seeds, the optimized prompts improve held‑out citation reward from 0.702 to 0.781, with a 95% confidence interval of [0.051, 0.107]. The study shows that the gain is driven by decomposed per‑dimension feedback rather than a single scalar reward, and that the improvement persists even with an out‑of‑family verifier and a hand‑written version of the discovered recipe.

By Elias Lumer, Sahil Sen, Vamse Kumar Subbiah, Kevin Paul
arXiv AI
Oct 26

RSI-Opt: Recursive Self-Improvement Optimizer Agents For Model Training

RSI-Opt introduces Recursive Self-Improvement Optimizer Agents that evolve their own training policies by accumulating knowledge from previous runs, including prompts, playbooks, optimizer code, and decision cadences. A meta‑agent rewrites and archives these policies, enabling reusable procedures and code that adapt to new tasks. Across multiple generations and tasks, RSI‑Opt improves optimizer performance, achieving higher z‑scores on evolution suites, better cross‑model transfer, and significant error reductions on PDE benchmarks compared to fixed‑prompt baselines.

By Binghang Lu, Yige Sun, Guang Lin
arXiv AI
Oct 16

Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine

Terminal‑RSI is a new framework for terminal agents that alternates Recursive Self‑Rewrite (RSR) of verified trajectories with Recursive Policy‑Refine (RPR) that inherits a critic. It introduces TB‑GAE for token‑specific baselines, Family Dense Reward (FDR) and Success‑Prioritized Reward (SPR) to calibrate progress, and uses a warm‑up critic, recursive handoff, and stage‑specific learning rates to coordinate the two refinement phases. The five‑stage pipeline applied to Qwen3.8‑27B produces Terminal‑RSI‑27B, achieving 19.84 % on Terminal‑Bench 4.0, 86.52 % on Terminal‑Bench 2.1, 16.95 % on Terminal‑Bench 3.0, and 79.00 % on Terminal‑Bench Hard, while improving process reward and strict completion on LHTB.

By Junyao Yang, Zongxia Li, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Ruiwen Zhou, Haitao Mi, Shuicheng Yan, Leowei Liang
arXiv AI
Jul 7

Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

arXiv:2607. 03702v1 Announce Type: new Abstract: Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals.

By Weiyang Guo, Zesheng Shi, Longhui Zhang, Zeen Zhu, Min Zhang, Jing Li