Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. While recent CIL advances have spurred significant interest across various modalities, the audio-visual setting remains underexplored.
The paper introduces a new Continual Audio‑Visual Segmentation (CAVS) task that enables continuous segmentation of new classes guided by audio. It identifies two key challenges—multi‑modal semantic drift and co‑occurrence confusion—and proposes a Collision‑based Multi‑modal Rehearsal (CMR) framework with Multi‑modal Sample Selection (MSS) and Collision‑based Sample Rehearsal (CSR) strategies to address them. Experiments on three audio‑visual incremental scenarios show that CMR outperforms single‑modal continual learning methods.
arXiv:2607. 00293v1 Announce Type: cross Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge.
arXiv:2607. 07907v1 Announce Type: cross Abstract: With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associations that originate from their training data.
The paper introduces a generalized multimodal foundation model that can handle arbitrary combinations of modalities and prediction tasks. It trains on large-scale synthetic multimodal datasets with diverse causal structures to learn transferable multimodal correlations. Experiments on 18 real-world datasets across 12 modalities and 11 tasks show competitive performance compared to specialized models without task-specific adaptation.
Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive. While recent token compression methods attempt to alleviate this burden, compressing modalities in isolation often destroys the temporal cross-modal anchors necessary for coherent reasoning.
The paper introduces PromptCCZSL, a framework that enables vision‑language models to continually learn new attributes, objects, and their unique compositions while avoiding forgetting. It uses a frozen VLM backbone with prompt‑based techniques, recency‑weighted multi‑teacher distillation, and several loss functions (CAL, OPL, IDL) to maintain prior knowledge and promote diverse, distinct embeddings. Experiments on UT‑Zappos and C‑GQA show significant performance gains over existing VLM‑based and non‑VLM baselines, establishing a new benchmark for continual compositional zero‑shot learning.
The paper introduces HASTEN, a hierarchical semantic tree anchoring framework that enhances CLIP-based class‑incremental learning by incorporating external hierarchical knowledge into hyperbolic space. By treating classes as part of a structured hierarchy rather than isolated labels, HASTEN stabilizes visual and textual feature mappings, reduces feature drift, and mitigates catastrophic forgetting across incremental tasks. Experiments on nine benchmarks demonstrate that HASTEN consistently outperforms existing methods.
arXiv:2608. 09227v1 Announce Type: new Abstract: Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive.
arXiv:2607. 10299v1 Announce Type: new Abstract: Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues.
ViSSFlow is a unified flow‑matching framework that jointly tackles Video‑to‑Sound (V2S) and Visual Text‑to‑Speech (VisualTTS) generation. It incorporates a Diffusion Transformer with a disentangled condition aggregation mechanism, using cross‑attention for semantic cues and self‑attention for dense visual information. The authors also introduce a feature‑level data synthesis strategy to compensate for limited joint training data, and demonstrate that a single model can be trained end‑to‑end without sacrificing performance on either task.
arXiv:2608. 04902v1 Announce Type: cross Abstract: Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis.