The paper introduces PIECE, a Parameter Importance-Driven Continual Learning method that selectively updates only 0.1% of core parameters to preserve general abilities while learning new domain knowledge. PIECE employs two importance estimators—PIECE‑F using Fisher Information and PIECE‑S combining gradient and curvature information—to guide updates. Experiments on three language models and two multimodal models demonstrate that PIECE maintains general capabilities and achieves state‑of‑the‑art continual learning performance without accessing prior training data or adding parameter overhead.
By Lingxiang Wang, Hainan Zhang, Zhiming Zheng
arXiv:2601. 19788v2 Announce Type: replace Abstract: Federated Continual Learning (FCL) leverages inter-client collaboration to better balance new knowledge acquisition and old knowledge retention on non-stationary data.
By Sixing Tan, Xianmin Liu
arXiv:2406.02447v5 Announce Type: replace
Abstract: Federated Learning (FL) aims at unburdening the training of deep models by distributing computation across multiple devices (clients) while safegua...
By Riccardo Salami, Pietro Buzzega, Matteo Mosconi, Mattia Verasani, Simone Calderara
arXiv:2606. 07500v1 Announce Type: cross Abstract: Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge.
By Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari
arXiv:2606. 31275v1 Announce Type: cross Abstract: Online Continual Self-Supervised Learning (OCSSL) aims to learn representations from a continuous stream of unlabeled data, without knowledge of task boundaries and under memory constraints.
By Julien Lefebvre, Stefan Duffner, Mathieu Lefort
arXiv:2605. 20247v2 Announce Type: replace-cross Abstract: Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs).
By Yang Liu, Toan Nguyen, Flora D. Salim
FederatedBytes is a small language model that processes text and multimodal data at the byte level, eliminating the need for tokenization. It uses a linear attention mechanism to keep memory and computational demands low, making it suitable for federated learning scenarios. Experiments on multilingual sentiment analysis tasks show that FederatedBytes outperforms models like ByT5 and LSTM, achieving higher accuracy while reducing memory and communication overheads.
By Ali Sarabadani, Massimo Piccardi, Mehran Abolhasan, Farzad Tofigh, Ian Zhou, Justin Lipman
The paper investigates how synthetic data used for fine‑tuning large language model (LLM) skill routers can lead to catastrophic forgetting of real and out‑of‑distribution (OOD) skills. By evaluating a production router with 34,396 skills, the authors show that synthetic‑data fine‑tuning improves in‑distribution retrieval but degrades performance on real and OOD data. They test several continual‑learning inspired mitigation methods—embedding‑anchor regularization, Learning without Forgetting, Elastic Weight Consolidation, and L2‑initialization—and find that these approaches both preserve OOD retrieval performance and boost synthetic in‑distribution retrieval by up to 13.98% for a 0.6B Qwen retriever and reranker.
By Syed Shariyar Murtaza, Yifan Nie, Utkarsh Soni, Eugene Wen, Arvid Frydenlund
arXiv:2607. 27665v1 Announce Type: new Abstract: Federated graph learning enables collaborative training over decentralized graph data without sharing raw graph information.
By Zekai Chen, Haodong Lu, Shihao Li, Weiwei Ji, Xunkai Li, Xun Wu, Yinlin Zhu, Rong-Hua Li
arXiv:2608. 00630v1 Announce Type: new Abstract: Achieving continual learning (CL) with deep neural networks requires balancing stability and plasticity while enabling knowledge transfer.
By Malavika Suresh, Ikechukwu Nkisi-Orji, Nirmalie Wiratunga
arXiv:2606. 15734v1 Announce Type: cross Abstract: Continual post-training enables models to absorb emerging knowledge after deployment, but repeatedly updating shared parameters can accumulate weight drift, potentially causing catastrophic forgetting and degrading general capabilities.
By Weihang Su, Jiacheng Kang, Jingyan Xu, Qingyao Ai, Jianming Long, Hanwen Zhang, Bangde Du, Xinyuan Cao, Min Zhang, Yiqun Liu
arXiv:2603. 12658v2 Announce Type: replace-cross Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs.
By Hongyang Chen, Zhongwu Sun, Hongfei Ye, Kunchi Li, Xuemin Lin