arXiv Machine Learning
Jul 31

Continual Learning with Vision-Language Models via Semantic-Geometry Preservation

arXiv:2603. 12055v3 Announce Type: replace-cross Abstract: Continual learning of pretrained vision-language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce geometric distortion.

By Chiyuan He, Zihuan Qiu, Fanman Meng, Runtong Zhang, Linfeng Xu, Qingbo Wu, Hongliang Li
arXiv Computer Vision
Aug 31

Dual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Models

The paper introduces Dual-Stream Semantic Guidance (DSSG), a framework for Source‑Fully‑Free Domain Adaptation of Vision‑Language Models that mitigates dual semantic drift through a caption stream and a class‑anchor stream. It adds a Dynamic Cross‑Modal Knowledge Distillation module and a Prototype Anchor Calibration extension (DSSG‑PAC) to reduce computation while maintaining performance. Experiments show DSSG outperforms state‑of‑the‑art methods and DSSG‑PAC cuts adaptation time by 18.9% with minimal loss in accuracy.

By Weiwei Xiang, Shun Peng, Guangyi Xiao, Hao Chen, Lei Yang
arXiv Computer Vision
Aug 27

A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training

The paper introduces a new task called Multimodal Unsupervised Continual Post-Training (MU‑CPT), which allows multimodal large language models (MLLMs) to continuously learn from streaming unlabeled data. It identifies token‑level visual dependence (VD) as essential for MU‑CPT, using its structural distortion to detect cross‑modal forgetting and its heterogeneity to guide new‑task learning. The proposed Visual Dependence‑Aware (VDA) framework includes Visually Constrained Optimal Transport (VC‑OT) to mitigate forgetting and Visually Modulated Adaptation (VMA) to enhance new‑task plasticity, achieving a balance between stability and adaptability in MU‑CPT.

By Kaichen Li, Zhilin Zhu, Jianhao Huang, Zhengqin Lai, Baochen Xiong, Zibo Shao, Yaguang Song, Linhui Xiao, Xiaoshan Yang, Changsheng Xu