← Back to all news
arXiv Machine Learning By Toan Nguyen, Yang Liu, Celso De Melo, Flora D. Salim

DynaTokens: Controlling Token Dynamics for Continual Video-Language Understanding

Read the original on arXiv Machine Learning →

The paper introduces DynaTokens, a transformer-based token generator that produces fine‑tuning tokens on demand for continual VideoQA with multimodal large language models. It uses shared generation weights and meta‑learning‑inspired regularisers to reduce task interference and forgetting, connecting the objective to sharpness‑aware optimisation for flatter cross‑task minima. Experiments on standard continual VideoQA benchmarks show that DynaTokens achieves higher average accuracy, lower forgetting, better zero‑shot generalisation, and robust cross‑modal transfer in a new ImageQA→VideoQA protocol.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jul 15

Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning

arXiv:2607. 12112v1 Announce Type: cross Abstract: Federated fine-tuning of Multimodal Large Language Models (MLLMs) across distributed networks enables privacy-sensitive adaptation to evolving data streams, yet a fundamental obstacle prevents robust deployment in dynamic environments: catastrophic forgetting, wherein sequential task updates erase previously acquired knowledge across visual, linguistic, and cross-modal representations.

By Jing Liu, Chenxuanyin Zou, Jiayang Ren, Gaoyun Fang, Chengfang Li, Yan Wang, Zhenchao Ma, Bo Hu
llmsragagentsfine-tuningmultimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Aug 27

A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training

The paper introduces a new task called Multimodal Unsupervised Continual Post-Training (MU‑CPT), which allows multimodal large language models (MLLMs) to continuously learn from streaming unlabeled data. It identifies token‑level visual dependence (VD) as essential for MU‑CPT, using its structural distortion to detect cross‑modal forgetting and its heterogeneity to guide new‑task learning. The proposed Visual Dependence‑Aware (VDA) framework includes Visually Constrained Optimal Transport (VC‑OT) to mitigate forgetting and Visually Modulated Adaptation (VMA) to enhance new‑task plasticity, achieving a balance between stability and adaptability in MU‑CPT.

By Kaichen Li, Zhilin Zhu, Jianhao Huang, Zhengqin Lai, Baochen Xiong, Zibo Shao, Yaguang Song, Linhui Xiao, Xiaoshan Yang, Changsheng Xu
multimodalsafety
More like this →