Data-DPO is a target model‑oriented supervised fine‑tuning data selection method that uses one‑step probing of the target model to generate pairwise data preferences, trains a lightweight reward model to capture these preferences, and then selects a training subset by combining target‑model preference, external quality scores, and marginal diversity. Experiments on Vision‑Flan and LLaVA‑CoT demonstrate that Data‑DPO consistently outperforms existing data selection baselines across multiple data budgets and even surpasses full data training performance.
By Peng Sun, Yi Yang, Antong Zhang, Chunxiao Li, Yanbo Wang, Dianbo Liu, xin chen, Kai Yu, Lu Chen, Tianfan Fu
arXiv:2501. 12147v2 Announce Type: replace-cross Abstract: Selecting appropriate training data is crucial for instruction fine-tuning of large language models (LLMs), which aims to (1) elicit strong capabilities, and (2) achieve balanced performance across different tasks.
By Qirun Dai, Dylan Zhang, Jiaqi W. Ma, Hao Peng
arXiv:2606. 18307v1 Announce Type: cross Abstract: Optimizing the training data distribution for Supervised Fine-Tuning (SFT) dictates the capability of Large Language Models (LLMs).
By Zefan Wang, Lincheng Li, Tianyu Yu, Yuan Yao
arXiv:2606. 09396v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation and often serves as the initialization stage for reinforcement learning (RL), but it can show weaker generalization than RL.
By Ke Wang, Shuangqi Li, Mathieu Salzmann, Pascal Frossard
arXiv:2509. 03647v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly serve as automated evaluators, yet they suffer from "self-preference bias": a tendency to favor their own outputs over those of other models.
By Dani Roytburg, Matthew Bozoukov, Matthew Nguyen, Jou Barzdukas, Simon Fu, Narmeen Oozeer
arXiv:2609.36659v1 Announce Type: new
Abstract: The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these st...
By Shufan Shen, Zhongni Hou, Junshu Sun, Yufei Zhang, Wei Lin, Guojun Yin, Qingming Huang, Shuhui Wang
arXiv:2606. 11189v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory.
By Tong Xie, Yuanhao Ban, Yunqi Hong, Sohyun An, Yihang Chen, Cho-Jui Hsieh
arXiv:2510. 06048v4 Announce Type: replace Abstract: Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks.
By Jie Hao, Rui Yu, Wei Zhang, Huixia Wang, Jie Xu, Mingrui Liu
arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.
By Hyung Gyu Rho
The paper proposes a two‑stage framework that first fine‑tunes a large language model (LLM) and then rectifies its outputs, allocating limited labeled data optimally between the stages. It argues that the usual mean‑squared‑error objective for fine‑tuning misaligns with the downstream rectification, and instead suggests minimizing prediction‑error variance for mean estimation or a scalarized variance metric for general M‑estimation. Empirical results confirm that this variance‑based fine‑tuning, combined with optimal data allocation, yields significant efficiency gains over using either fine‑tuning or rectification alone, or using the conventional objective.
By Zikun Ye, Jinglong Zhao, Lei Wang
arXiv:2606. 32034v1 Announce Type: cross Abstract: LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions.
By Sergio Hern\'andez-Guti\'errez, Matteo Merler, Ilze Amanda Auzina, Joschka Str\"uber, Ameya Prabhu, Matthias Bethge
RapidUn is a parameter reweighting framework that uses influence estimates to guide LoRA-only updates for efficient unlearning of targeted behaviors in large language models. It operates in a practical PEFT setting with a small forget set and limited retain buffer, converting cross-sample influence into fixed sample-specific weights for weighted LoRA unlearning. Experiments on Llama‑3‑8B with Dolly‑15k and Alpaca‑57k datasets show RapidUn achieves lower trigger ASR than Fisher, GA, and LoReUn while preserving clean utility, and delivers a 77× wall‑clock speedup over clean‑corpus LoRA retraining, with additional evaluations supporting its effectiveness.
By Guoshenghui Zhao, Huawei Lin, Weijie Zhao