arXiv Computation and Language October 5, 2026 By Baohang Li, Xiaocheng Feng, Yichong Huang, Chengpeng Fu, Wenshuai Huo, Zekun Zhou, Zekun Yuan, Tingjia Zhang, Bing Qin

Adaptive Mutual Distillation for Balanced Multi-Task Post-Training of Large Language Models

Read the original on arXiv Computation and Language →

The paper introduces Adaptive Mutual Distillation (AMD), a post‑training framework that jointly trains two large language models using different task‑balancing strategies. AMD evaluates and selects distillation weight adjustments via short training probes and task‑wise validation, leading to models that outperform supervised fine‑tuning baselines on six benchmarks across three backbones. Merging the two AMD models further improves performance, surpassing multi‑task fine‑tuning by an average of 2.91 points.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jul 7

Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe

arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.

By Wenjin Hou, Shangpin Peng, Weinong Wang, Zheng Ruan, Yue Zhang, Zhenglin Zhou, Mingqi Gao, Yifei Chen, Kaiqi Wang, Hongming Yang, Chengquan Zhang, Zhuotao Tian, Han Hu, Yi Yang, Fei Wu, Hehe Fan

llms efficiency multimodal benchmarks

Hugging Face Trending Papers

Jun 22

Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model.

llms fine-tuning efficiency

arXiv Computation and Language

Aug 21

EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptation

arXiv:2604. 26170v2 Announce Type: replace Abstract: Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge.

By Ting-Wei Li, Sirui Chen, Jiaru Zou, Yingbing Huang, Tianxin Wei, Jingrui He, Hanghang Tong

llms benchmarks safety

arXiv Machine Learning

Sep 22

CAMFT: Conflict-Aware Mergeable Fine-Tuning for Large Language Models

CAMFT is a Conflict‑Aware Mergeable Fine‑Tuning method designed to make task adaptation efficient and merge‑aware for large language models. Unlike existing approaches that only resolve parameter conflicts after fine‑tuning, CAMFT shapes mergeability during training by guiding each task to update sparse coordinates with lower cross‑task conflict. Experiments show that CAMFT outperforms standard fine‑tuning baselines in multi‑task merging scenarios.

By Jingang Zhou, Haiyang Guo, Yuan Ma, Han Zhu, Xu-Yao Zhang

llms fine-tuning

Hugging Face Trending Papers

Aug 24

Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Data

Structured data exists in many forms (tables, knowledge graphs, charts, and time series), and converting it into text may involve different generation tasks. However, most prior work on data-to-text (...

llms fine-tuning efficiency benchmarks

arXiv AI

Jun 18

From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging

arXiv:2506. 14126v2 Announce Type: replace-cross Abstract: Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets.

By Stefan Horoi, Guy Wolf, Eugene Belilovsky, Gintare Karolina Dziugaite

fine-tuning