arXiv Computation and Language September 1, 2026 By Huiyi Zhang, Zijian Li, Xiaocheng Feng, Weitao Ma, Xiaoliang Yang, Yichong Huang, Bing Qin

ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

Read the original on arXiv Computation and Language →

The paper introduces ACTD, an Anchor-Based Cross-Tokenizer Distillation method that aligns vocabularies and sequences to transfer reasoning capabilities from large language models to smaller students. It uses a novel anchor loss with residual regularization to reduce alignment noise and extends the approach to multiple teachers. Experiments on five reasoning benchmarks with three teachers show state‑of‑the‑art results, with the multi‑teacher variant outperforming existing baselines.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Jun 9

Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.

By Yifan Niu, Han Xiao, Dongyi Liu, Zelong Wang, Dihong Gong, Yasheng Wang, Jia Li

llms nlp fine-tuning efficiency benchmarks

arXiv AI

Jun 19

Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts

arXiv:2402. 14035v4 Announce Type: replace-cross Abstract: Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality.

By Zichang Liu, Qingyun Liu, Yuening Li, Liang Liu, Anshumali Shrivastava, Shuchao Bi, Lichan Hong, Ed H. Chi, Zhe Zhao

llms efficiency

arXiv AI

Jun 19

Reinforcement-aware Knowledge Distillation for LLM Reasoning

arXiv:2602. 22495v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students.

By Zhaoyang Zhang, Shuli Jiang, Yantao Shen, Yuting Zhang, Dhananjay Ram, Shuo Yang, Zhuowen Tu, Wei Xia, Stefano Soatto

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 30

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.

By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jul 17

Decoupled Alignment for Robust Plug-and-Play Adaptation

arXiv:2406. 01514v4 Announce Type: replace-cross Abstract: We introduce a training-free safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning or reinforcement learning from human feedback.

By Haozheng Luo, Jiahao Yu, Wenxin Zhang, Jialong Li, Chenghao Qiu, Yimin Wang, Eric Hanchen Jiang, Jerry Yao-Chieh Hu, Yan Chen, Binghui Wang, Xinyu Xing, Han Liu

llms reinforcement-learning fine-tuning efficiency safety

arXiv AI

Jun 4

Invariant Gradient Alignment for Robust Reasoning Distillation

arXiv:2606. 05025v1 Announce Type: cross Abstract: Large language models (LLMs) suffer from shortcut learning: they systematically fail on out-of-distribution (OOD) inputs whose semantic surface differs from training data, even when the logical structure is identical.

By Zehua Cheng, Wei Dai, Jiahao Sun

llms fine-tuning efficiency benchmarks safety

One email a morning, machine-written

Related stories

Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts

Reinforcement-aware Knowledge Distillation for LLM Reasoning

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

Decoupled Alignment for Robust Plug-and-Play Adaptation

Invariant Gradient Alignment for Robust Reasoning Distillation