Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,102 stories · RSS feed

arXiv Machine Learning

Sep 10

Decoder-Side Semantic Conditioning for Low-Bitrate Neural Speech Compression

arXiv:2512.21653v2 Announce Type: replace-cross Abstract: Speech codecs are usually optimized for waveform fidelity, allocating bits to acoustic detail that can be inferred from linguistic structure....

By Liuyang Bai, Weiyi Lu, Li Guo

efficiency multimodal

arXiv Machine Learning

Sep 10

Interleaved Speech Language Models Latently Work In Text

arXiv:2606.22473v2 Announce Type: replace-cross Abstract: Speech language models (SLMs) increasingly combine speech and text, often by interleaving their tokens within a single sequence. Yet how thes...

By Talia Sternberg, Gallil Maimon, Yossi Adi

llms diffusion multimodal

arXiv Computation and Language

Sep 10

BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Models

arXiv:2609.09554v1 Announce Type: new Abstract: We introduce BuzzASR, a collection of language-specialized fine-tuned Whisper models adapted for automatic speech recognition (ASR) in 102 languages. L...

By Shivam Singh, Aditya Yadavalli, Catherine Arnett, Alex Warstadt

llms nlp fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 10

False positive bias in AI-powered speech-based cognitive screening for multilingual English speakers in the UK

arXiv:2602.13047v2 Announce Type: replace Abstract: Conversational speech reveals early signs of cognitive decline, including dementia and mild cognitive impairment (MCI). AI models show promise for...

By Madhurananda Pahar, Caitlin Illingworth, Dorota Braun, Bahman Mirheidari, Lise Sproson, Daniel Blackburn, Heidi Christensen

agents multimodal safety

arXiv Computer Vision

Sep 10

Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence

arXiv:2609.09184v1 Announce Type: new Abstract: Vision-language model (VLM) confidence may change in aggregate when visual evidence is degraded while remaining structurally inconsistent within indivi...

By Muhamathu Ameer Ali Aacaas Muhamath

llms multimodal

arXiv Computer Vision

Sep 10

Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gap

arXiv:2609.09417v1 Announce Type: new Abstract: Vision-language models (VLMs) show promise for agricultural classification, but zero-shot performance on disease, pest, damage, quality, and species id...

By Earl Ranario, Jared Smith, Lars Lundqvist, Urmil Jatin Chandarana

llms multimodal benchmarks

arXiv Computer Vision

Sep 10

VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models

arXiv:2609.09396v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric...

By Zaid Pervaiz Bhat, Nimra Nayyar, Arihant Jain, Lap Fung Chan, John Suchanek, Yu Wang, Varun Praveen, Tomasz Kornuta, Vidya Nariyambut Murali

llms nlp robotics multimodal benchmarks safety

arXiv Computer Vision

Sep 10

UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport

arXiv:2609.10224v1 Announce Type: new Abstract: Vision-language models such as CLIP embed images and text in a shared space, where modality-specific distributions often remain separated. Existing acc...

By Zonglin Yang, Huilan Ma, Xudan Zheng, Yuejun Xie

llms rag multimodal safety

arXiv Computer Vision

Sep 10

Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs

arXiv:2609.10333v1 Announce Type: new Abstract: Uncertainty estimation for medical vision--language models (VLMs) using conformal prediction has gained increasing attention due to its distribution-fr...

By Xuan Cuong Ngo, Ngan Le

llms fine-tuning multimodal safety

arXiv Computer Vision

Sep 10

Identifying Habit, Physics, and Nuisance in Robot World Models

arXiv:2609.09210v1 Announce Type: cross Abstract: Teleoperated demonstrations are often multimodal even when the underlying dynamics are nearly deterministic given the executed action. We argue that...

By Jinting Hang, Zhenhui Cai

diffusion robotics multimodal benchmarks

arXiv Computer Vision

Sep 10

Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition

arXiv:2609.10394v1 Announce Type: cross Abstract: Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the...

By Rishabh Jain, Aristeidis Papadopoulos, Zhaofeng Lin, Naomi Harte

multimodal benchmarks

arXiv Computer Vision

Sep 10

Synergistic Vision-Language Reinforcement Enables Scalable On-Demand Analysis across Diverse Clinical Tasks

arXiv:2505.03380v2 Announce Type: replace Abstract: Accurate delineation of tumors and surrounding organs-at-risk is essential for radiotherapy, surgery and treatment response assessment, yet remains...

By Haonan Wang, Jiaji Mao, Lehan Wang, Qixiang Zhang, Marawan Elbatel, Yi Qin, Huijun Hu, Baoxun Li, Wenhui Deng, Weifeng Qin, Hongrui Li, Jialin Liang, Jun Shen, Xiaomeng Li

llms computer-vision multimodal

arXiv Computer Vision

Sep 10

Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes

arXiv:2512.14177v4 Announce Type: replace Abstract: Large Vision-Language Models (LVLMs) often produce plausible but unreliable outputs, making robust uncertainty estimation essential. Recent work on...

By Joseph Hoche, Andrei Bursuc, David Brellmann, Gilles Louppe, Pavel Izmailov, Angela Yao, Gianni Franchi

llms rag computer-vision multimodal benchmarks

arXiv Computer Vision

Sep 10

RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture

arXiv:2601.15891v4 Announce Type: replace Abstract: Vision-language pretraining has driven progress in medical image representation learning, but it depends on paired image-text data and can inherit...

By Anas Anwarul Haq Khan, Mariam Husain, Pratik Jalan, Kshitij Jadhav

rag computer-vision multimodal safety

arXiv Computer Vision

Sep 10

Let ViT Speak: Generative Language-Image Pre-training

arXiv:2605.00809v3 Announce Type: replace Abstract: In this paper, we present \textbf{Gen}erative \textbf{L}anguage-\textbf{I}mage \textbf{P}re-training (GenLIP), a minimalist generative pretraining...

By Yan Fang, Mengcheng Lan, Zilong Huang, Weixian Lei, Yunqing Zhao, Yujie Zhong, Yingchen Yu, Qi She, Yao Zhao, Yunchao Wei

llms computer-vision multimodal benchmarks

arXiv Computer Vision

Sep 10

Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perception

arXiv:2606.06714v3 Announce Type: replace Abstract: Human perception of surface slant from texture exhibits systematic, graded biases that emerge reliably in psychophysical experiments. Prior work sh...

By Qian Zhang, Michal Golovanevsky, Fulvio Domini, James Tompkin

llms fine-tuning multimodal benchmarks

arXiv Computer Vision

Sep 10

Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs

arXiv:2608.05592v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) have made strong progress in video understanding, yet long videos remain difficult: the visual token budge...

By Ziling Huang, Shin'ichi Satoh

llms multimodal benchmarks

arXiv AI

Sep 10

Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation

arXiv:2609.08275v1 Announce Type: new Abstract: Recent multi-shot audio-video generators can produce increasingly coherent and cinematic outputs, but coherence does not imply the ability to execute e...

By Tianyi Zeng, Junchao Liao, Yujie Wei, Ziying Zhang, Litao Li, Tianyi Wang, Zhichao Wei, Shuyao Xu, Wenwen Qiang, Siyu Zhu, Zhenghao Zhang, Long Qin

agents diffusion nlp multimodal benchmarks safety

arXiv Computer Vision

Sep 10

No Free Checker: A Survey of Verifiers for Robot Policies

arXiv:2609.09250v1 Announce Type: cross Abstract: A verifier for robot policies reads a candidate behavior and returns a score for how well it did, used both to evaluate vision-language-action polici...

By Yang Wan, Xihang Yue, Zhirui Liu, Ziyuan Chu, Shuxun Wang, Yuhan Chen, Xiaonan Jiang, Xukun Zhu, Yubo Dong, Linchao Zhu

robotics multimodal safety

arXiv AI

Sep 10

Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Models

arXiv:2609.05540v1 Announce Type: cross Abstract: Many medical conditions require diagnosis through detailed, multi-context clinical assessment rather than from visual appearance alone. Despite this,...

By Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel, Hansa Meghwani, Jyotika Singh, Ranjeet Gupta, Graham Horwood, Tao Sheng, Avi Sil, Sujith Ravi, Dan Roth

llms multimodal safety

Get the daily briefing