← Back to all news
arXiv Machine Learning By Jiacheng Wang, Zhijie Liu, Ping Jian, Zirong Chen, Ke Ren Liao, Zhen Yang, Zhongbin Guo

Patch the Distribution Mismatch: RL Rewriting Agent for Stable Off-Policy SFT

Read the original on arXiv Machine Learning →

The paper introduces a reinforcement‑learning based rewriting agent that rewrites training data to reduce the distribution mismatch between supervised fine‑tuning (SFT) and a language model’s generation distribution. By formulating data rewriting as a policy‑learning problem, the authors train a lightweight LoRA rewriting policy that optimizes alignment with question‑answering style, maintains semantic diversity, and enforces task consistency. Experiments on three instruction‑tuned backbones show that models fine‑tuned with the rewritten data achieve downstream performance comparable to standard SFT while mitigating degradation on non‑downstream benchmarks, and preliminary tests suggest the policy can transfer across domains such as logical reasoning and medical question answering.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jul 21

From Evidence to Trajectory: Abductive Reasoning Path Synthesis for Retrieval-Augmented Generation Agents Development

arXiv:2509. 23071v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) agent development is hindered by the lack of executable ground-truth agent-environment interaction trajectories.

By Muzhi Li, Jinhu Qi, Yihong Wu, Minghao Zhao, Liheng Ma, Yifan Li, Xinyu Wang, Zhenghan Tai, Zixing Song, Yingxue Zhang, Ho-fung Leung, Irwin King
llmsragagentsnlpreinforcement-learningfine-tuningbenchmarks
More like this →
arXiv Machine Learning
Sep 4

DE-Venus: A Data-Efficient RLVR Framework for Large Language Models

DE‑Venus is a unified, data‑efficient framework for reinforcement learning with verifiable rewards (RLVR) tailored to large language models. It structures the RLVR lifecycle into three modules—Active Data Selection, Weak Supervision Construction, and Training‑Time Supervision Refinement—allowing method‑specific decisions to be expressed as dataset transitions or online transformations while maintaining distributed execution contracts. Experiments on public benchmarks and three business scenarios show that DE‑Venus can preserve or improve model quality using only 10% of labels or 13% of relevant data, and can cut convergence steps by 63%–75% in selected business configurations.

By Shenzhi Yang, Guangcheng Zhu, Kai Tang, Zhengqing Zang, Xing Zheng, Haobo Wang, Yingfan Ma, Bowen Song, Bo Han, Bo An, Lei Feng, Weiqiang Wang, Junbo Zhao, Gang Chen
llmsreinforcement-learningbenchmarks
More like this →
arXiv Computation and Language
Aug 28

Boosting LLM Exploration via Weak-Model Guidance in RLVR

The paper introduces a method to enhance large language model (LLM) exploration in Reinforcement Learning with Verifiable Rewards (RLVR) by guiding the target model with partial reasoning trajectories from smaller, weaker language models. This weak-model guidance disrupts over‑confidence, preserves generative diversity, and mitigates entropy collapse without extra fine‑tuning or complex reward designs. Experiments on mathematical benchmarks show consistent improvements over vanilla RLVR, especially as the number of allowed attempts ($k$) increases, indicating broader reasoning coverage.

By Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
llmsreinforcement-learningfine-tuningbenchmarks
More like this →