arXiv AI
Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Jul 13
Active Offline-to-Online Reinforcement Learning
Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction. This offline-to-online RL (O2O-RL) paradigm is particularly promising in nonstationary domains where interaction is costly or potentially hazardous.
arXiv Machine Learning
Jul 16
Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback
arXiv:2607. 13389v1 Announce Type: new Abstract: Reinforcement Learning (RL) post-training is increasingly used to adapt foundation models for reasoning, planning, and feedback-driven robot-learning pipelines, but constrained post-training resources are often summarized by a single total FLOP budget.
arXiv AI
Jul 14
Active Offline-to-Online Reinforcement Learning
arXiv:2607. 11720v1 Announce Type: cross Abstract: Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction.
arXiv Machine Learning
Sep 14
MInTRL: Off-policy Intervention can boost On-policy RL
MInTRL (Minimal Intervention Reinforcement Learning) expands exploration in on-policy reinforcement learning by inserting sparse, local corrections into rollouts via a judge-intervention policy. These interventions replace erroneous suffixes and immediately return control to the main policy, allowing the agent to explore beyond its natural trajectory while maintaining on-policy data. The method uses a sequence-level advantage-regression objective, avoiding importance sampling, and demonstrates superior performance on math and code benchmarks compared to standard on-policy and off-policy baselines.
arXiv AI
Oct 9
Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation
The paper proposes a method for allocating limited counterfactual annotation resources from multiple sources—such as domain experts and large language models—to improve off-policy evaluation in contextual bandits. By formulating an integer optimization problem that balances source-specific costs and error rates, the authors derive a majorization‑minimization algorithm that iteratively reduces estimator variance. Experiments on synthetic clinical and education bandit data demonstrate that the optimized allocation cuts mean‑squared error by over 20% compared to no annotation.
arXiv AI
Aug 19
Task Specialization Fine-Tuning for Contextual Reinforcement Learning
The paper introduces Task Specialization Fine-Tuning (TSFT), an online framework that allocates a limited fine‑tuning budget across multiple task regions in Contextual Reinforcement Learning. TSFT predicts fine‑tuning performance with a simple parametric model and solves the budget allocation problem exactly using integer linear programming. Experiments on combinatorial optimization, continuous control, and LLM fine‑tuning show that TSFT outperforms baselines in task coverage and approaches oracle performance.