arXiv AI
Beyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty Estimation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 27
Cross-Attentive Bayesian Low-Rank Adaptation for Multimodal Uncertainty Estimation
arXiv:2604.16657v2 Announce Type: replace-cross Abstract: Parameter-efficient fine-tuning (PEFT) enables efficient adaptation of frozen language models, but most PEFT methods remain deterministic or...
arXiv Computer Vision
Sep 30
Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment
arXiv:2501.18157v2 Announce Type: replace-cross Abstract: Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions fr...
arXiv Machine Learning
Sep 23
Unified Multimodal Uncertain Inference
Unified Multimodal Uncertain Inference (UMUI) is a new task that requires models to generate calibrated probability estimates for hypotheses conditioned on premises across text, audio, and video modalities. The authors create a human‑annotated evaluation set with scalar probability judgments for audio, visual, and audiovisual settings, and benchmark their approach on existing text and audio datasets. Their CLUE framework, which blends self‑consistent teacher calibration with distribution‑based confidence probing, enables a 3B‑parameter model to match or surpass zero‑shot baselines up to 32B parameters across all modalities.
arXiv Computer Vision
Oct 21
Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis
The paper introduces TAME, a Text‑Anchored Modality Experts framework that treats language as a fixed, always‑active expert and routes audio‑textual and video‑textual cross‑modal branches as dynamic auxiliary experts. It incorporates lightweight Prophet Heads to estimate sample‑level uncertainty for each modality, guiding a reliability‑aware routing mechanism for expert aggregation. Experiments on CMU‑MOSI, CMU‑MOSEI, and CH‑SIMS show that TAME achieves state‑of‑the‑art performance while reducing parameters by up to 78.7% and FLOPs by up to 41.2% compared to a mixture‑of‑experts baseline.
arXiv Machine Learning
Jun 3
Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals
arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.
arXiv AI
Oct 16
Hierarchical Interpretable Multitask Audio Representation Learning: An Analysis of Event-Scene Relationships through the Information Bottleneck
The paper proposes a hierarchical conditioning approach for joint Acoustic Scene Classification and Audio Tagging, predicting scene categories directly from intermediate event probabilities. Extensive experiments on three backbones and varied loss‑weighting settings show that this method matches or surpasses traditional multi‑head models while enabling synchronized dual‑task learning for certain beta values. An interpretability framework combining information bottleneck analysis, manifold geometry, and Formal Concept Analysis on TF‑IDF event representations demonstrates that hierarchical conditioning reduces conditional entropy, increases mutual information, and improves latent cluster separability, yielding a more structured and semantically faithful feature space.