arXiv AI

ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training

ReDraft is a reference‑driven revision method for continual post‑training of large multimodal language models. It uses the model’s own incorrect outputs as references, revises them, verifies the revisions, and fine‑tunes on the accepted ones, thereby combining explicit supervision with policy proximity. On tasks such as Counting, Clock Reading, and Jigsaw, ReDraft outperforms standard supervised fine‑tuning and on‑policy methods, achieving higher target‑task gains while dramatically reducing forgetting.

Hugging Face Trending Papers
Oct 5

Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning

The paper demonstrates that off‑policy merging, called grafting, outperforms on‑policy self‑distillation for continual learning. Grafting learns updates on an earlier donor checkpoint, scales the update, and optionally masks sensitive directions, thereby reducing interference with existing capabilities. Across various continual learning scenarios, grafting achieves better new‑task and old‑task performance without costly on‑policy sampling.

arXiv Machine Learning
Oct 23

How to Teach Large Multimodal Models New Skills

The paper investigates how to add new skills to large multimodal models (LMMs) without erasing previously learned abilities. By sequentially fine‑tuning on five target skills and tracking performance on eight held‑out benchmarks across three model families, the authors discover that forgetting on one skill can partially recover when the model is later tuned on a different skill. They attribute this to shifts in the output token distribution and propose two simple, selective tuning recipes—updating only self‑attention projection layers or only the MLP Gate&Up while freezing the Down projection—that achieve a better learning‑forgiveness trade‑off than full‑LLM tuning or other common forgetting‑mitigation methods. whyItMatters":"The study shows that carefully choosing which model components to fine‑tune can enable LMMs to acquire new skills while preserving prior knowledge, offering a simpler and more effective alternative to existing forgetting‑mitigation techniques."

By Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem
arXiv Machine Learning
Oct 22

Combating Catastrophic Forgetting in Continual Domain Adaptation via Knowledge Recasting

The paper introduces Skill Schema Transport (SST), a post‑training framework that mitigates catastrophic forgetting in continual domain adaptation by translating new domain knowledge into a model’s existing skill space. SST abstracts each example type into a reusable skill schema, preserving essential operations while removing domain‑specific surface forms, thereby turning knowledge updates into procedural reuse rather than trajectory memorization. Experiments on two 3B instruction‑tuned backbones show that SST consistently reduces forgetting and improves generalization, achieving up to a 5.75‑point performance gain and a dramatic drop in maximum forgetting from 42.41 to 7.10 points on the Qwen‑3B MATH stream.

By Zijie Liu, Zhen Tan, Charles Fleming, Tianlong Chen
arXiv AI
Aug 24

Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillation

The paper identifies a specific issue in supervised fine‑tuning (SFT) of large language models called factual access failure, where models can recognize correct facts under constrained tests but fail to generate them in open‑ended settings. It demonstrates that SFT can cause both genuine wrong answers and expression‑level errors such as verbosity or formatting mismatches. To mitigate this, the authors propose Recall‑Anchored Distillation (RAD), a self‑distillation method that aligns the fine‑tuned model with the base model’s soft output distribution on unlabeled out‑of‑distribution text, thereby recovering lost factual recall without needing labeled data.

By Haodong Chen, Yadong Wang, Shengtao Wen, Dong Liang, Xiang Chen
arXiv Machine Learning
Oct 16

Not All Change Is Damage: Update-Matched Unlearning for Large Language Models

The paper introduces Update‑Matched Unlearning (UMU), a method that applies the same simulated relearning update to both a defended model and an undefended reference to better preserve plasticity while resisting relearning of deleted knowledge. Experiments on Llama and Qwen models with TOFU and FaithUn datasets show that UMU reduces harmful and benign relearning by 30.4 and 9.9 points on average and increases new‑task accuracy by 3.5 points, supporting the reference‑timing hypothesis. The study demonstrates that robust unlearning should selectively prevent recovery of deleted knowledge without suppressing the model’s ability to learn new tasks.

By Jie Li, Youming Tao, Bowei He, Meng Ding
arXiv Machine Learning
Nov 2

Steering Vectors as a Training Signal in LLM Post-Training

Steering vectors (SVs) are residual-stream directions that can shift LLM behavior when added to hidden states during inference. This study investigates whether the same SVs can guide parameter updates during post‑training, exploring supervised learning (SFT and distillation) and reinforcement learning with verifiable rewards (RLVR). The authors find that SV injection improves performance over plain SFT and distillation, achieving up to a 12% gain in certain distillation settings, and that applying a correctness‑contrast direction as advantage repair in RL recovers learning signals for held‑out math benchmarks. They also identify failure boundaries related to code generation, cross‑family layer transfer, and inference‑time application on SV‑trained checkpoints.

By Tiejin Chen, Maunil Vyas, Huaiyuan Yao, Hua Wei