arXiv AI

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

arXiv:2606. 23885v1 Announce Type: cross Abstract: Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder.

arXiv AI

Jun 4

A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks

arXiv:2510. 24342v2 Announce Type: replace Abstract: Prior brain-AI alignment studies are typically constrained by specific inputs and tasks, limiting their ability to capture organizational properties across models with different modalities.

By Silin Chen, Yuzhong Chen, Caiwei Wang, Zifan Wang, Junhao Wang, Zifeng Jia, Keith M Kendrick, Tuo Zhang, Lin Zhao, Dezhong Yao, Tianming Liu, Xi Jiang

llms fine-tuning multimodal safety

arXiv AI

Jun 30

SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

arXiv:2602. 23353v2 Announce Type: replace-cross Abstract: The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world.

By Simon Roschmann, Paul Krzakala, Sonia Mazelet, Quentin Bouniot, Zeynep Akata

llms rag multimodal safety

arXiv AI

Jun 24

When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models

arXiv:2605. 08245v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) increasingly power high-stakes applications, from medical imaging to autonomous systems, yet they routinely hallucinate, confidently describing content not present in the input.

By Harshvardhan Saini, Samyak Jha, Yiming Tang, Dianbo Liu

llms rag agents fine-tuning multimodal benchmarks safety

arXiv AI

Jun 30

Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation

arXiv:2606. 29464v1 Announce Type: cross Abstract: Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs that can efficiently train contrastive vision-language models under strict data and compute budgets.

By Jongoh Jeong, Sun-Kyung Lee, Kuk-Jin Yoon

llms rag efficiency multimodal benchmarks safety

arXiv AI

Jun 3

Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

arXiv:2604. 18572v2 Announce Type: replace-cross Abstract: The Platonic Representation Hypothesis suggests that neural networks trained on different modalities (e.

By A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, Alexei A. Efros

llms multimodal safety

arXiv AI

Jun 16

Attention, not scale, drives human-AI alignment in multimodal language prediction

arXiv:2308. 06035v4 Announce Type: replace Abstract: Humans routinely draw on visual context to predict upcoming words.

By Viktor Kewenig, Andrew Lampinen, Samuel A. Nastase, Christopher Edwards, Quitterie Lacome D'Elascombe, Akilles Rechardt, Jeremy I Skipper, Gabriella Vigliocco

llms multimodal benchmarks safety

arXiv AI

Jun 8

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

arXiv:2602. 07026v3 Announce Type: replace-cross Abstract: Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions.

By Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

llms rag fine-tuning multimodal safety

arXiv AI

Jul 1

ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs

arXiv:2606. 31054v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image.

By Zhiyuan Yao, Zheren Fu, Zhixiao Zheng, Jiajun Li, Yi Tu, Zhendong Mao

llms diffusion reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jul 3

Transformer Geometry Observatory TGO-II: Representational Similarity Observatory

arXiv:2607. 02386v1 Announce Type: cross Abstract: While Vision Transformers have achieved remarkable success across computer vision and language applications, the geometric evolution of their internal representations throughout training remains insufficiently understood.

By Kaustubh Kapil, Kishor P. Upla

llms computer-vision safety

arXiv AI

Jun 3

Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.

By Xinpeng Dong, Min Zhang, Kairong Han, Xu Tan, Fei Wu, Kun Kuang

llms computer-vision multimodal benchmarks safety

arXiv Machine Learning

Aug 4

Hierarchical Pre-Training of Vision Encoders with Large Language Model

arXiv:2604. 00086v2 Announce Type: replace-cross Abstract: The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks.

By Eugene Lee, Ting-Yu Chang, Jui-Huang Tsai, Jiajie Diao, Chen-Yi Lee

llms rag computer-vision multimodal benchmarks safety

arXiv AI

Jun 30

Multimodal Representation Alignment for Cross-modal Information Retrieval

arXiv:2506. 08774v2 Announce Type: replace-cross Abstract: Different machine learning models can represent the same underlying concept in different ways.

By Fan Xu, Luis A. Leiva

llms rag multimodal benchmarks safety

Related stories