Optimizing Anytime Reasoning via Budget Relative Policy Optimization
Read the original on arXiv Machine Learning →The paper introduces AnytimeReasoner, a framework that optimizes large language models for anytime reasoning by truncating the reasoning process to fit sampled token budgets and requiring the model to summarize the answer for verification. This approach adds verifiable dense rewards, improving credit assignment in reinforcement learning. The authors also propose Budget Relative Policy Optimization (BRPO), a variance reduction technique that enhances learning robustness and efficiency, and demonstrate that their method outperforms GRPO on mathematical reasoning tasks across various budgets and prior distributions.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.