arXiv AI

Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

arXiv:2607. 02909v1 Announce Type: cross Abstract: Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language.

arXiv Computer Vision

Sep 22

Hierarchical Prompt Learning for Hyperbolic Vision-Language Models

arXiv:2609.24276v1 Announce Type: new Abstract: Hyperbolic vision-language models (VLMs) represent image and text features in a geometry naturally suited to hierarchy, but their adaptation to downstr...

By Andro Erdelez, Pascal Mettes, Behzad Bozorgtabar

llms rag multimodal benchmarks

arXiv AI

Aug 11

Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings

arXiv:2603. 26798v2 Announce Type: replace-cross Abstract: Vision-language model (VLM) encoders such as CLIP enable strong retrieval and zero-shot classification in a shared image-text embedding space, yet the semantic organization of this space is rarely inspected.

By Gesina Schwalbe, Mert Keser, Moritz Bayerkuhnlein, Edgar Heinert, Annika M\"utze, Marvin Keller, Sparsh Tiwari, Georgii Mikriukov, Diedrich Wolter, Jae Hee Lee, Matthias Rottmann

llms rag multimodal safety

arXiv AI

Jul 13

H3Former: Hypergraph-based Semantic-Aware Aggregation via Hyperbolic Hierarchical Contrastive Loss for Fine-Grained Visual Classification

arXiv:2511. 10260v2 Announce Type: replace-cross Abstract: Fine-Grained Visual Classification (FGVC) remains a challenging task due to subtle inter-class differences and large intra-class variations.

By Yongji Zhang, Siqi Li, Kuiyang Huang, Yue Gao, Yu Jiang

rag benchmarks

arXiv AI

Sep 18

Lens: Bringing the Right Semantic Perspective into Focus for Training-Free Multimodal Representation Learning

The paper introduces Lens, a training‑free framework that aligns multimodal representations with the semantic perspective required by downstream tasks. Lens uses a task‑specific readout phrase to anchor the perspective and then aggregates token states after the full input, ensuring the extracted representation reflects task‑conditioned evidence integration rather than generic salient content. The method achieves a Precision@1 of 63.9 across 36 MMEB datasets, outperforming the nearest training‑free baseline by 10.2 points.

By Xinran Liu, Shouqian Shi, Yixian Chen, Ruizhi Chen, Xin-Wei Yao, Sheng Zhong

llms rag multimodal

arXiv Machine Learning

Jul 15

Visual Species Recognition with Large Multimodal Models as Post-Hoc Correctors

arXiv:2512. 15748v2 Announce Type: replace Abstract: Visual Species Recognition (VSR) is a fundamental task in scientific disciplines that require species-level identification, including ecology, palynology, evolutionary biology, systematics, and phylogenetics.

By Tian Liu, Anwesha Basu, James Caverlee, Shu Kong

llms multimodal benchmarks

arXiv AI

Sep 1

Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

arXiv:2608.29313v1 Announce Type: cross Abstract: CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embed...

By Matin Mahmood, Antonio Rueda-Toicen, Mohamed ElBassat, Seifeldin Elkerdany, Weixing Wang, Gerard de Melo

llms rag multimodal safety

arXiv AI

Sep 16

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

arXiv:2609.16722v1 Announce Type: new Abstract: Scaling Multimodal Large Language Models (MLLMs) to long-form video understanding is bottlenecked by the explosion of visual tokens, which saturates co...

By Haoyu Guo, Yuan Feng, Junlin Lv, Mingjun Xiao, S Kevin Zhou, Xike Xie

llms multimodal

arXiv AI

4d ago

LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension

arXiv:2609.37426v1 Announce Type: cross Abstract: Modern vision-language models (VLMs) have shown promising results in long-video understanding due to the rich semantic information they can capture....

By Arka Mukherjee, Kaleen Shrestha, Larissa Zhu, Maja Matari\'c

llms rag agents multimodal benchmarks

arXiv AI

Jun 3

Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

arXiv:2605. 18160v2 Announce Type: replace-cross Abstract: In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information.

By Xinpeng Dong, Min Zhang, Kairong Han, Xu Tan, Fei Wu, Kun Kuang

llms computer-vision multimodal benchmarks safety

arXiv AI

Jul 21

LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.

By Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang

llms rag multimodal benchmarks

arXiv AI

Jun 12

Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality

arXiv:2606. 13288v1 Announce Type: cross Abstract: Contrastively trained vision-language models like CLIP, have made remarkable progress in learning joint image-text representations, but still face challenges in compositional understanding.

By Wei Li, Zhen Huang, Xinmei Tian

llms diffusion multimodal benchmarks

arXiv AI

Jun 16

Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs

arXiv:2606. 16193v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their internal visual representations remain difficult to interpret.

By Yusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, Hao Wang

llms multimodal benchmarks safety

Related stories