← Back to all news
arXiv AI By Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama

Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models

Read the original on arXiv AI →

The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 11

Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages

arXiv:2603. 12658v2 Announce Type: replace-cross Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs.

By Hongyang Chen, Zhongwu Sun, Hongfei Ye, Kunchi Li, Xuemin Lin
llmsfine-tuningbenchmarkssafety
More like this →
arXiv Machine Learning
Aug 28

Unifying Detection and Adaptation in Task-Free Continual Learning

The paper introduces FiUni, a Fisher-guided unified framework that performs batch-level task detection and parameter-efficient continual adaptation for large language models. By exploiting orthogonality in the Fisher information matrix’s Kronecker-Factored Approximate Curvature subspaces, FiUni constructs frozen subspaces to guide low-rank adaptation and matches incoming batch windows to historical subspaces. This approach allows the model to decide whether to reuse, expand, or create new subspaces, balancing knowledge sharing and task isolation while achieving competitive performance with fewer trainable parameters.

By Dezheng Han, Anbang Zhang, Zhihao Zhu, Shuaishuai Guo
llmsfine-tuning
More like this →