arXiv AI
FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 16
Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback
arXiv:2607. 13389v1 Announce Type: new Abstract: Reinforcement Learning (RL) post-training is increasingly used to adapt foundation models for reasoning, planning, and feedback-driven robot-learning pipelines, but constrained post-training resources are often summarized by a single total FLOP budget.
arXiv AI
Sep 1
BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning
BCPPO is a new variant of Proximal Policy Optimization that uses Bachelier-inspired cost‑prediction networks to generate a smooth penalty based on disagreement among critics. The method keeps temporal‑difference learning unchanged, applies a saturation‑aware controller to manage cost penalties, and deploys only the policy network. Across extensive experiments, BCPPO outperforms comparators in achieving higher mean returns while maintaining lower or comparable CVaR in all tested tasks.
arXiv AI
Sep 4
FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience
arXiv:2609. 03241v1 Announce Type: cross Abstract: A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode.
arXiv Machine Learning
1d ago
CERO: Where and When to Allocate Rollouts for RL Post-Training
arXiv:2610.09679v1 Announce Type: new Abstract: Adaptive rollout methods for group-relative reinforcement learning typically allocate a fixed per-update budget across prompts. We instead study how to...
arXiv Machine Learning
Sep 25
Certified Predictive Value-of-Advice Gating for Cost-Aware Language-Model Guidance in Reinforcement Learning
The paper proposes a method for selectively querying language‑model advice in reinforcement learning by predicting the value of potential responses and only querying when the expected benefit outweighs the cost. It introduces a certified, response‑contingent metareasoning framework that guarantees near‑optimal advice usage under certain assumptions, and demonstrates that a calibrated controller with Qwen2.5 advisors can improve task performance while drastically reducing the number of advice calls on the BabyAI benchmark.
arXiv AI
Sep 16
Learning-Guided Planning in Large Dynamic Action Spaces: Budgeted Tree Search for One-to-Many Mobile Charging
The paper introduces LP‑BTS, a learning‑guided planning framework for mobile charging in large, dynamic action spaces. It uses a graph proposal policy to narrow candidate stops, a value critic to evaluate leaf nodes, and edge‑budgeted PUCT to compare short simulated futures before action selection. Experiments on a 30‑scenario battery‑life benchmark show LP‑BTS achieving the highest survival and alive‑AUC, outperforming domain‑engineered baselines and heuristic policies.