arXiv Machine Learning

Multimodal Concept Bottleneck Models

arXiv:2606. 19882v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts.

arXiv AI

Jun 2

Hoeffding Concept Bottleneck Models with Applications to Overhead Images

arXiv:2606. 00082v1 Announce Type: cross Abstract: Explainability of deep learning algorithms is critical for computer-vision applications with high-stake decisions.

By Cl\'ement B\'enard, Manon Arfib, Christophe Labreuche, Victor Qu\'etu

computer-vision

arXiv AI

Jun 30

On the Faithfulness of Post-Hoc Concept Bottleneck Models

arXiv:2606. 30498v1 Announce Type: cross Abstract: Human decision-making interprets the world through high-level concepts, such as recognizing a bird by its belly color.

By Laines Schmalwasser, Jan Blunk, Niklas Penzel, Julia Niebling, Joachim Denzler

llms multimodal benchmarks

arXiv Machine Learning

Aug 12

Clarity: The Flexibility-Interpretability Trade-Off in Sparsity-aware Concept Bottleneck Models

arXiv:2601. 21944v3 Announce Type: replace Abstract: The widespread adoption of deep learning models in computer vision has intensified concerns about interpretability.

By Konstantinos P. Panousis, Diego Marcos

computer-vision efficiency multimodal safety

arXiv AI

Jun 12

Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

arXiv:2606. 13288v1 Announce Type: cross Abstract: Contrastively trained vision-language models like CLIP, have made remarkable progress in learning joint image-text representations, but still face challenges in compositional understanding.

By Wei Li, Zhen Huang, Xinmei Tian

llms diffusion multimodal benchmarks

arXiv AI

Jun 26

Bridging Vision and Language Concepts through Optimal Transport Semantic Flow

arXiv:2606. 26891v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) promise transparent reasoning by predicting through human-interpretable concepts, yet their effectiveness fundamentally depends on how well visual and textual representations are aligned or matched.

By Chenyang Zhang, Anqi Dong, Guangming Zhu, Nuoye Xiong, Siyuan Wang, Lin Mei, Liang Zhang

diffusion multimodal safety

arXiv AI

Jun 8

MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

arXiv:2606. 06696v1 Announce Type: cross Abstract: Vision and language models (VLMs) hold immense promise to transform biomedical imaging workflows, from detecting lesions in chest X-rays to profiling cellular features in microscopy.

By Ryan D'Cunha, Alejandro Lozano, Xiaoxiao Sun, Daniel Vela Jarquin, Min Woo Sun, Josiah Aklilu, James Burgess, Yuhui Zhang, Ryan Nayebi, Paola Avila, Robayo, Jin Ye, Ming Hu, Zhongying Deng, Junjun He, Xin Chen, Yue Yao, Robert Tibshirani, Jeffrey J. Nirschl, Serena Yeung-Levy

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Jul 10

A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding

arXiv:2512. 21414v2 Announce Type: replace-cross Abstract: Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools.

By Christina Liu, Alan Q. Wang, Joy Hsu, Jiajun Wu, Ehsan Adeli

llms multimodal benchmarks

arXiv Machine Learning

Jun 30

Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders

arXiv:2606. 29888v1 Announce Type: new Abstract: Vision-language models map images and text into a joint embedding space.

By Chungpa Lee, Jihoon Kwon, Kyle Min, Jy-yong Sohn

llms rag multimodal safety

arXiv AI

Jul 7

TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models

arXiv:2607. 04593v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated impressive capabilities across different tasks, but their computational cost is dominated by the large number of visual tokens fed to the language model.

By Riccardo Renzulli, Gabriele Spadaro, Shruthi Gowda, Alaa Eddine Mazouz, Van-Tam Nguyen

llms diffusion fine-tuning efficiency multimodal benchmarks

arXiv AI

Jul 29

Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

arXiv:2607. 25527v1 Announce Type: cross Abstract: Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities.

By Weiming Zhuang, Jiabo Huang, Jingtao Li, Zhizhong Li, Chen Chen, Sina Sajadmanesh, Lingjuan Lyu

llms multimodal benchmarks

arXiv AI

Jun 24

MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning

arXiv:2606. 22220v2 Announce Type: replace-cross Abstract: Memorization in machine learning models enables high performance on rare in-distribution samples by capturing their atypical patterns.

By Wenhao Wang, Franziska Boenisch, Michael Backes, Adam Dziedzic

multimodal

arXiv AI

Jun 24

Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations

arXiv:2606. 24716v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) are increasingly used to extract interpretable concepts from vision and vision language models, yet existing evaluation methods largely rely on proxy metrics or qualitative inspection rather than measuring semantic correspondence.

By Jonas Klotz, Cassio F. Dantas, Pallavi Jain, Diego Marcos, Beg\"um Demir

llms rag benchmarks safety

Related stories