arXiv AI

Semifactual Credit-Augmented Policy Optimization

arXiv Machine Learning
Nov 2

Optimizing Anytime Reasoning via Budget Relative Policy Optimization

The paper introduces AnytimeReasoner, a framework that optimizes large language models for anytime reasoning by truncating the reasoning process to fit sampled token budgets and requiring the model to summarize the answer for verification. This approach adds verifiable dense rewards, improving credit assignment in reinforcement learning. The authors also propose Budget Relative Policy Optimization (BRPO), a variance reduction technique that enhances learning robustness and efficiency, and demonstrate that their method outperforms GRPO on mathematical reasoning tasks across various budgets and prior distributions.

By Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, Min Lin
arXiv Computation and Language
6d ago

BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

BALTO is a Balanced Token-level Policy Optimization framework designed to reduce hallucinations in large language models. It extracts factual claims, verifies them against reference context, and assigns token-level credit in a balanced way to shift probability mass toward faithful content. Experiments on ConFiQA, RAGTruth, and FinLLM-Eval show BALTO outperforms existing baselines in faithfulness metrics and improves the faithfulness–informativeness trade‑off.

By Ning Li, Zixuan Guo, Yan Xu, Wenbo Fei, Yifan Niu, Chang Luo, Yasheng Wang, Weiwen Liu, Yong Yu, Weinan Zhang
arXiv AI
Oct 15

Verifiable Process Rewards for Agentic Reasoning

The paper introduces Verifiable Process Rewards (VPR), a framework that uses symbolic or algorithmic task-solving oracles as action-level verifiers to provide dense rewards for intermediate actions in agentic reasoning tasks. VPR is applied to Sokoban, Sudoku, and Minesweeper, and improves policy performance over outcome-reward baselines. The authors also demonstrate VPR’s effectiveness in a tool‑use benchmark, highlighting its reliance on reliable task oracles and the difficulty of extending it to less structured environments.

By Huining Yuan, Zelai Xu, Huaijie Wang, Xiangmin Yi, Jiaxuan Gao, Xiao-Ping Zhang, Yu Wang, Chao Yu, Yi Wu
arXiv AI
Jun 18

Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards

arXiv:2606. 18810v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps.

By Yingyu Shan, Yuhang Guo, Zihao Cheng, Zeming Liu, Xiangrong Zhu, Xinyi Wang, Jiashu Yao, Wei Lin, Hongru Wang, Heyan Huang
arXiv AI
Sep 30

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

SIPO (Self‑Instructing Policy Optimization) unifies reinforcement learning with on‑policy self‑distillation by using a contrastive self‑teacher to generate token‑level credit signals. The method samples multiple rollouts per prompt, pairs each with a reference answer and its mistakes, and uses the difference in teacher log‑probabilities to provide dense feedback while still respecting the overall task reward. Experiments on reasoning and code‑generation benchmarks show that SIPO outperforms both RLVR and OPSD baselines without requiring an external teacher or extra generation steps.

By Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu, Fengran Mo, Jinghan Zhang, Mung Yao Jia, Gyuseok Lee, Yang Zhang, Na Wei, Dong Wang