← Back to all news
arXiv Machine Learning By Haiquan Wen, Yiwei He, Bei Peng, Guangliang Cheng

Online Self-Weighted Fine-Tuning

Read the original on arXiv Machine Learning →

Online Self-Weighted Fine‑Tuning (OSW‑FT) augments standard supervised fine‑tuning by adding online, trajectory‑level weighting: for each query the model estimates its current success rate from a small number of inference‑only rollouts and rescales the SFT loss accordingly. The method keeps the optimization direction anchored to the expert trajectory while adapting the update magnitude online, and it is shown to be unbiased for any finite rollout count with a convergence analysis. Across Qwen3 models from 0.6B to 4B, OSW‑FT consistently outperforms plain SFT on challenging benchmarks such as AIME, achieving a favorable compute‑performance trade‑off with only two online rollouts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 10

ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search

ReST‑RL introduces a unified Reinforced Self‑Training (ReST) policy‑value framework that enhances large language model (LLM) reasoning by combining an optimized ReST‑style GRPO algorithm with a value‑guided search (VM‑MCTS). The ReST‑GRPO component reshapes trajectory distributions to increase reward variance and expose policies to more informative partial states, improving training efficiency. VM‑MCTS trains a Value Model from self‑collected Monte‑Carlo Tree Search targets and uses it during inference to provide precise process signals and verification scores, boosting reasoning accuracy across coding benchmarks and out‑of‑domain math and science tasks.

By Sining Zhoubian, Dan Zhang, Jie Tang
llmsreinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Aug 12

Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge

arXiv:2604. 02621v2 Announce Type: replace-cross Abstract: Reinforcement Learning (RL) substantially improves the reasoning capabilities of language models, but most existing RL fine-tuning approaches rely entirely on ground-truth verifiable rewards and thus labeled datasets with verifiable answers.

By Yiyang Shen, Lifu Tu, Weiran Wang
llmsreinforcement-learningfine-tuningefficiency
More like this →
arXiv Machine Learning
Jul 10

HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning

arXiv:2601. 22448v2 Announce Type: replace Abstract: RLVR has become a standard recipe for training LLMs on reasoning tasks with verifiable outcomes, but when rollout generation dominates the cost, efficiency hinges on which prompts are sampled and when.

By Weiqi Wang, Xin Liu, Binxuan Huang, Hejie Cui, Rongzhi Zhang, Changlong Yu, Shuowei Jin, Jingfeng Yang, Qingyu Yin, Zhengyang Wang, Zheng Li, Yifan Gao, Priyanka Nigam, Bing Yin, Lihong Li, Yangqiu Song
llmsreinforcement-learningbenchmarks
More like this →