Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,406 stories · RSS feed

arXiv AI

Jun 8

GP-Adapter: Gaussian Process CLIP-Adapter for Few-Shot Out-of-Distribution Detection

arXiv:2606. 07102v1 Announce Type: cross Abstract: We propose GP-Adapter, a training-free framework that augments CLIP (Contrastive Language-Image Pre-training) with Gaussian Process (GP) uncertainty modeling for few-shot classification and out-of-distribution (OOD) detection.

By Taisei Saito, Koretaka Ogata, Takafumi Hiroi

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jun 8

TokaMind: A Multi-Modal Transformer Foundation Model for Tokamak Plasma Dynamics

arXiv:2602. 15084v2 Announce Type: replace-cross Abstract: We present TokaMind, to our knowledge the first open-source foundation model for tokamak plasma dynamics, based on a Multi-Modal Transformer (MMT) and pretrained on heterogeneous diagnostics from the publicly available MAST dataset.

By Tobia Boschi, Andrea Loreti, Nicola C. Amorisco, Rodrigo H. Ordonez-Hurtado, C\'ecile Rousseau, George K. Holt, Eszter Sz\'ekely, Alexander Whittle, Samuel Jackson, Adriano Agnello, Stanislas Pamela, Alessandra Pascale, Robert Akers, Juan Bernabe Moreno, Vassil Alexandrov, Mykhaylo Zayats

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jun 8

MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

arXiv:2606. 06696v1 Announce Type: cross Abstract: Vision and language models (VLMs) hold immense promise to transform biomedical imaging workflows, from detecting lesions in chest X-rays to profiling cellular features in microscopy.

By Ryan D'Cunha, Alejandro Lozano, Xiaoxiao Sun, Daniel Vela Jarquin, Min Woo Sun, Josiah Aklilu, James Burgess, Yuhui Zhang, Ryan Nayebi, Paola Avila, Robayo, Jin Ye, Ming Hu, Zhongying Deng, Junjun He, Xin Chen, Yue Yao, Robert Tibshirani, Jeffrey J. Nirschl, Serena Yeung-Levy

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Jun 8

AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

arXiv:2512. 13278v2 Announce Type: replace-cross Abstract: Agentic reinforcement learning has advanced large language models (LLMs) to reason through long chain-of-thought trajectories while interleaving external tool use.

By Jiaru Zou, Ling Yang, Yunzhe Qi, Sirui Chen, Mengting Ai, Ke Shen, Jingrui He, Mengdi Wang

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 8

FLOWREADER: Min-Cost Flow Optimization for Multi-Modal Long Document Q&A

arXiv:2606. 07235v1 Announce Type: cross Abstract: Long, multimodal documents force retrieval-augmented systems to assemble answers from evidence fragmented across text, tables, and slides broken across cells in a long table, spread over multiple slides, or split between a figure and its discussion.

By Ambuj Mehrish, Sebatiano Vascon

rag multimodal

arXiv AI

Jun 8

MatterDoor: Sampling Zero-shot Spatio-semantic Priors using Generative Models

arXiv:2510. 11014v2 Announce Type: replace-cross Abstract: Autonomous robots often view rooms only partially, through a doorway, where the walls and scene structure hide the geometry and task-relevant semantics needed for safe navigation and goal-directed action.

By Subhransu S. Bhattacharjee, Hao Lu, Dylan Campbell, Rahul Shome

agents computer-vision robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 8

Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation

arXiv:2606. 06836v1 Announce Type: cross Abstract: Language-guided UAV agents must execute long-horizon semantic instructions while producing smooth, physically feasible continuous flight commands, yet existing Vision-Language Navigation (VLN) benchmarks typically use discrete or coarse actions and existing UAV Vision-Language-Action (VLA) tasks focus on short, atomic maneuvers.

By Xiangyi Zheng, Xiangyu Wang, Qinan Liao, Zimu Tang, Yue Liao, Dongyue Lyu, Guodong Wang, Junjie Liu, Si Liu

llms agents diffusion multimodal benchmarks

arXiv AI

Jun 8

A robust PPG foundation model using multimodal physiological supervision

arXiv:2606. 07365v1 Announce Type: cross Abstract: Photoplethysmography (PPG), a non-invasive measure of changes in blood volume, is widely used in both wearable devices and clinical settings.

By Eloy Geenjaar, Vince Calhoun, Scott Daly, Gouthaman KV, Lie Lu, Trisha Mittal, Daniel P. Darcy

multimodal benchmarks

arXiv AI

Jun 8

What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Media

arXiv:2606. 06784v1 Announce Type: cross Abstract: Public social media posts can reveal private information through weak cues scattered across text, images, or metadata.

By Zifan Peng, Yini Huang, Aiwen Lu, Qiming Ye, Peixian Zhang, Jingyi Zheng, Yule Liu, Xuechao Wang, Xinlei He, Jiaheng Wei

agents multimodal benchmarks safety

arXiv AI

Jun 8

Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization

arXiv:2606. 07033v1 Announce Type: new Abstract: Open-vocabulary audio-visual event localization (OV-AVEL) jointly models audio-visual cues to recognize and temporally localize events, including categories unseen during training.

By Zhe Yang, Ruyi Zhang, Hongtao Chen, Wenrui Li, Hengyu Man, Wangmeng Zuo, Xiaopeng Fan

multimodal benchmarks safety

arXiv AI

Jun 8

MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

arXiv:2605. 05225v3 Announce Type: replace-cross Abstract: Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect.

By Bo Li, Chuan Wu, Shaolin Zhu

llms multimodal benchmarks

arXiv AI

Jun 8

HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec

arXiv:2606. 06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models.

By Arjun Gangwar, S Umesh

llms multimodal

arXiv AI

Jun 8

MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval

arXiv:2604. 09552v2 Announce Type: replace-cross Abstract: Engineering rulebooks and technical standards contain multimodal information like dense text, tables, and illustrations that are challenging for retrieval augmented generation (RAG) systems.

By Kiarash Naghavi Khanghah, Hoang Anh Nguyen, Anna C. Doris, Amir Mohammad Vahedi, Daniele Grandi, Faez Ahmed, Hongyi Xu

llms rag agents nlp multimodal benchmarks

arXiv Machine Learning

Jun 8

Diagnosing Visual Ignorance in Vision-Language Models

arXiv:2606. 06890v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) frequently rely on language priors, producing confident answers that are weakly grounded in visual evidence.

By Runyu Zhou, Qi Zhang, Qixun Wang, Yisen Wang

llms multimodal benchmarks

arXiv AI

Jun 8

Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models

arXiv:2606. 06534v1 Announce Type: cross Abstract: Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point.

By Jialin Wu, Qianru Zhang, Georges El Fakhri, Xiaofeng Liu

llms nlp multimodal benchmarks safety

arXiv AI

Jun 8

SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

arXiv:2606. 06943v1 Announce Type: cross Abstract: Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations.

By Sunoh Kim, Daeho Um

llms efficiency multimodal benchmarks safety

arXiv AI

Jun 8

MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models

arXiv:2606. 06853v1 Announce Type: cross Abstract: The new era has witnessed a remarkable capability to extend Vision-Language Models (VLMs) for tackling tasks of video understanding.

By Yifan Xu, Chao Zhang, Ruifei Ma, Fei Gao, Zhifei Yang, Jiaxing Qi, Zhipeng Chen

llms agents diffusion multimodal benchmarks safety

Hugging Face Trending Papers

Jun 8

ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China

We introduce ChinaHeritaQA, a multimodal benchmark dataset for evaluating the cultural reasoning abilities of vision-language models (VLMs) on UNESCO World Heritage sites in China. The dataset comprises 2,279 in-the-wild images paired with 14,133 bilingual (Chinese/English) multiple-choice QA pairs spanning seven cognitive dimensions, from basic identity recognition to historical periodization and architectural analysis.

llms nlp multimodal benchmarks

arXiv AI

Jun 6

CLEAR: Cognition and Latent Evaluation for Adaptive Routing in End-to-End Autonomous Driving

arXiv:2606. 06219v1 Announce Type: cross Abstract: End-to-end autonomous driving models often struggle to balance multi-modal maneuver generation with real-time inference constraints.

By Yining Xing, Zehong Ke, Zhiyuan Liu, Yanbo Jiang, Wenhao Yu, Jianqiang Wang

agents diffusion robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jun 6

MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation

arXiv:2605. 28579v2 Announce Type: replace Abstract: Large language models (LLMs) have recently advanced text-driven 3D generation, yet Text-to-CAD remains far from supporting industrial product design.

By Xiaoyu Dong, Zhi Li, Xiao-Ming Wu

llms multimodal benchmarks safety

Get the daily briefing