Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

7,971 stories · RSS feed

arXiv AI

Sep 30

GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets

arXiv:2609.37798v1 Announce Type: cross Abstract: Speech self-supervised learning aims to learn general-purpose representations for downstream speech tasks. However, current approaches rely on comple...

By Gaspard Bott\'e, S\'everin Baroudi, Samir Sadok, Francesco Paissan, Thomas Hueber, Xavier Alameda-Pineda, Ricard Marxer, Mirco Ravanelli

multimodal

arXiv AI

Sep 30

It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

arXiv:2609.37863v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model r...

By Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem, Lotem Peled-Cohen

llms multimodal

arXiv AI

Sep 30

$S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient

arXiv:2609.37976v1 Announce Type: cross Abstract: LLMs trained with Chain-of-thought excel in reasoning capability, but often come with excessive token cost. We find that the core of reasoning capaci...

By Hongbo Ma, Sansheng Cao, Jiajun Fan, Bangji Yang, Ge Liu

llms multimodal

arXiv AI

Sep 30

FloCA: Towards Faithful and Logically Consistent Flowchart Reasoning

arXiv:2602.14035v2 Announce Type: replace Abstract: Flowchart-oriented dialogue (FOD) systems aim to guide users through multi-turn decision-making or operational procedures by following a domain-spe...

By Jinzi Zou, Bolin Wang, Shuo Zhang, Nuo Xu, Junzhou Zhao

llms agents multimodal safety

arXiv AI

Sep 30

Spatial Memory Agent: Experience-Grounded Procedural Memory for Spatial Intelligence

arXiv:2608.12743v3 Announce Type: replace Abstract: Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ab...

By Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du, Shengtao Zhang, Zhiyue Zhao, Yuling Xi, Hao Chen

agents reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Sep 30

Privacy-Preserving Full-Body Meshing from mmWave Radar via Mesh Foundation Model Supervision

arXiv:2609.34768v2 Announce Type: replace Abstract: Millimeter-wave (mmWave) radar enables privacy-preserving human perception, but the extreme sparsity of point clouds from commercial single-chip se...

By Shuxing Zhang, Yongquan Ni, Zhenyu Ding, Yawen Lin

llms efficiency multimodal benchmarks safety

arXiv AI

Sep 30

COMiT: Learning Structured Visual Tokens through Sequential Communication

arXiv:2602.20731v2 Announce Type: replace-cross Abstract: Discrete image tokenizers provide a sequential interface for vision and multimodal models, but are typically optimized for reconstruction or...

By Aram Davtyan, Yusuf Sahin, Yasaman Haghighi, Sebastian Stapf, Pablo Acuaviva, Alexandre Alahi, Paolo Favaro

llms nlp multimodal safety

arXiv AI

Sep 30

Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness

arXiv:2606.18363v3 Announce Type: replace-cross Abstract: Language models trained on large-scale vision-language data have demonstrated strong potential for embodied agents. Harnessing models through...

By Haowen Liu, Xirui Li, Shaoxiong Yao, Peng Shi, Tianyi Zhou, Jia-Bin Huang, Furong Huang, Jiayuan Mao

llms agents robotics multimodal

arXiv AI

Sep 30

Trustworthiness Costs of Domain Adaptation in Small Language Models:A Cross-Architecture Empirical Study

arXiv:2608.00042v2 Announce Type: replace-cross Abstract: Domain adaptation of small language models (SLMs) has emerged as a practical strategy for deploying capable NLP systems in resource-constrain...

By Ramesh B. Paramkusham

llms nlp reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Sep 30

Diffusion Reward Models

arXiv:2609.33803v2 Announce Type: replace-cross Abstract: Reward models underpin the alignment of large language models, yet the dominant designs reduce each prompt--response pair to a point estimate...

By Xiangyang Wang, Bingxiang He, Zeyuan Liu, Jiaze Wang, Ziqing Qiao, Yuxin Zuo, Huan-ang Gao, Cheng Qian, Wenbin Zhang, Ran Li, Youbang Sun, Ning Ding, Yuanchun Shi, Zhiyuan Liu, Chaojun Xiao, Chun Yu

llms diffusion reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Sep 30

Physical Cross-Modal Masked Autoencoding for Seismic-to-Well Representation Learning

arXiv:2609.36193v1 Announce Type: new Abstract: Learning from scientific measurements often requires aligning modalities with different spatial support and resolution. Subsurface characterization is...

By Meher Gajula, Keyla Gonzalez, Ben Lasscock, Alejandro Valenciano

rag multimodal

arXiv Machine Learning

Sep 30

Variational Mixtures and Multi-Marginal Flow Matching: Advancing Statistical Inference with Biological Applications

arXiv:2609.36911v1 Announce Type: new Abstract: In this thesis I develop methods for statistical inference when the distributions arising from complex biological systems are multi-modal, geometricall...

By Oskar Kviman

diffusion multimodal

arXiv Machine Learning

Sep 30

Scaling Full Conformal Image Classifiers

arXiv:2609.37298v1 Announce Type: new Abstract: Conformal prediction provides set-valued predictions with distribution-free coverage guarantees, making it attractive for high-stakes image classificat...

By Julio Silva-Rodr\'iguez, Ender Konukoglu

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Sep 30

Reactive Real-Time Flow Policies via Asynchronous Distribution Alignment

arXiv:2609.36540v1 Announce Type: cross Abstract: Generalist robot policies such as vision-language-action models (VLAs) have achieved remarkable generalization, but their inference delays can confli...

By Moritz Zoellner, Reece O'Mahoney, Ioannis Havoutis, Rohan Paleja

robotics efficiency multimodal safety

arXiv AI

Sep 30

TReVS: Integrating Textual Relevance and Visual Saliency for Efficient Vision-Language Model Token Pruning

arXiv:2609.37581v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) excel at visual understanding and reasoning but often incur substantial inference costs due to the large number of visu...

By Jing Wang, Zhiping Wu, Dongdong Ren, Youfang Han, Wei Zhao, Wenbin Li

llms efficiency multimodal benchmarks