arXiv AI
Dynamic Important Example Mining for Reinforcement Finetuning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 2
Finetuning with Sampling: SFT Learns Better Than You Think
The paper introduces a Markov chain Monte Carlo (MCMC) sampling algorithm that transforms off‑policy traces into more on‑policy data for supervised finetuning (SFT). By tailoring the data distribution rather than changing the learning objective, the method allows SFT to match or surpass traditional post‑training techniques such as reinforcement learning across tasks like scientific skill acquisition, mathematical reasoning, and open‑ended expertise. The resulting models generalize better, forget less, and exhibit strong distributional performance, demonstrating that sampling can serve as a model‑native operator for broader post‑training applications.
arXiv Machine Learning
Oct 3
Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
The paper investigates how reusing past samples can improve the sample efficiency of Proximal Policy Optimization (PPO). Two variants, wPPO-U and wPPO-BH, are introduced within a multiple importance weighting framework, each reusing data from recent iterations while preserving core PPO mechanics. The authors derive theoretical policy improvement bounds for both variants and empirically evaluate their impact on continuous control tasks.
Hugging Face Trending Papers
Jun 2
Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines. However, current reward evaluation relies on heterogeneous criteria such as rule-based verifiers, ground-truth references, procedural checklists, and complex rubrics, where a unified mechanism to integrate all types of evidence remains unexplored.
arXiv Machine Learning
Sep 14
MInTRL: Off-policy Intervention can boost On-policy RL
MInTRL (Minimal Intervention Reinforcement Learning) expands exploration in on-policy reinforcement learning by inserting sparse, local corrections into rollouts via a judge-intervention policy. These interventions replace erroneous suffixes and immediately return control to the main policy, allowing the agent to explore beyond its natural trajectory while maintaining on-policy data. The method uses a sequence-level advantage-regression objective, avoiding importance sampling, and demonstrates superior performance on math and code benchmarks compared to standard on-policy and off-policy baselines.
arXiv AI
Jul 7
Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
arXiv:2607. 04728v1 Announce Type: cross Abstract: Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data.
arXiv AI
Jul 14
Active Offline-to-Online Reinforcement Learning
arXiv:2607. 11720v1 Announce Type: cross Abstract: Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction.