The paper investigates whether auxiliary trajectory preferences can enhance small-language-model self‑improvement and transfer when only limited supervision is available. Two reward schemes—confidence‑based (SC) and entropy‑based (UID‑Conj.)—are compared through online reward augmentation and offline training on AIME problems. UID‑Conj. yields a 2.08‑point macro‑average improvement on the adaptation benchmark and outperforms uniform weighting in offline weighted SFT by 1.93 points, with task‑dependent benefits for SC and UID‑Conj. rewards.
arXiv:2507.21931v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks....
The paper introduces Cross-Behavior Policy Optimization (CBPO), a method that unifies offline demonstrations and policy-generated rollouts into a single regression loss using a fixed-reference target. CBPO eliminates the need for separate supervised fine‑tuning and reinforcement learning objectives, allowing positive and negative offline completions, reference responses, and current and historical rollouts to share a common reward‑labeled format and replay buffer. Experiments on two model scales and four mathematical reasoning benchmarks show that CBPO, especially the CBPO‑Unified variant, outperforms baseline methods in avg@32 and pass@32 metrics, with historical rollouts providing additional gains.
arXiv:2608. 07531v1 Announce Type: cross Abstract: Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence.
arXiv:2607. 25600v1 Announce Type: cross Abstract: Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation.
CounterRoute is an online reinforcement‑learning framework that jointly learns how to route a language model’s reasoning between a ‘think’ and a ‘direct answer’ mode, using a single shared policy derived from a dual‑mode checkpoint. It employs counterfactual rollouts to credit routing decisions and a curriculum that starts with forced dual‑mode rollouts before shifting to self‑routed updates, achieving better accuracy‑efficiency trade‑offs across nine benchmarks. The method reduces generated tokens by up to 51% on Qwen3‑8B while improving macro‑average accuracy, and its routing strategy generalizes to unseen coding, science, and commonsense tasks.
arXiv:2608.23493v1 Announce Type: new Abstract: Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However,...
Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation. We investigate whether verbalized confidence from black-box language models can serve as an actionable signal for retrieval routing.
arXiv:2606. 30345v1 Announce Type: cross Abstract: Enabling large language models to achieve stable self-improvement without external expert supervision remains a central challenge in complex reasoning tasks.
Agentic Critical Training (ACT) is a reinforcement learning approach that trains language‑model agents to judge actions directly by pairing expert actions with alternatives and rewarding correct selection. Unlike imitation learning or self‑reflection, ACT does not require reference rationales and can be reused across model sizes. Experiments on Qwen3‑8B and Olmo‑3‑7B‑Instruct show that ACT improves performance on ALFWorld, WebShop, ScienceWorld, and outperforms supervised reflection baselines, with significant gains over IL and IL→RL baselines.
TalentRL is a strategy‑guided reinforcement learning framework that routes natural‑language skills hierarchically, using coarse‑grained talents to guide fine‑grained skill retrieval. The method trains talent routing via reward contrasts and optimizes execution with group‑relative advantages, achieving significant performance gains on open‑domain QA benchmarks and agentic tasks such as ALFWorld and WebShop.
arXiv:2604. 10688v2 Announce Type: replace-cross Abstract: On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult.
Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Mo...