Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,018 stories · RSS feed

arXiv AI

Jul 28

Sheaf-Laplacian Obstruction and Projection Hardness for Cross-Modal Compatibility on a Modality-Independent Site

arXiv:2604. 07632v2 Announce Type: replace-cross Abstract: Cross-modal representations vary in how easily they can be aligned, and compatibility is generally non-transitive: two modalities may align through an intermediate modality at lower complexity than through a direct map.

By Tibor Sloboda

multimodal

arXiv AI

Jul 28

Cheap Probes Predict Expensive Training in 3D-CT Vision--Language Models

arXiv:2607. 22771v1 Announce Type: cross Abstract: Picking the frozen image encoder for a 3D~CT vision--language model (VLM), together with the token-compression scheme on top of it, is a search over many candidates.

By Renjie Liang

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jul 28

Novel Claim or D\'ej\`a Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking

arXiv:2607. 23514v1 Announce Type: cross Abstract: Multimodal automated fact-checking (MAFC) verifies claims by retrieving and reasoning over external evidence.

By Haorui He, Xinwen Chen, Dacheng Wen, Reynold Cheng, Francis C. M. Lau, Yupeng Li

llms multimodal benchmarks

arXiv AI

Jul 28

A Few Words Go a Long Way: Language Guided Robot Policy Synthesis

arXiv:2607. 23784v1 Announce Type: cross Abstract: While vision-language-action models have demonstrated impressive zero-shot manipulation capabilities, they remain fundamentally black box policies that are difficult to interpret, adapt, or correct when they inevitably fail.

By Daphne Chen, Archit Ritesh Jain, Eric Goossen, Emma Romig, Michael Murray, Nick Walker, Maya Cakmak

llms agents robotics multimodal benchmarks

arXiv Machine Learning

Jul 28

LanteRn: Latent Visual Structured Reasoning

arXiv:2603. 25629v2 Announce Type: replace-cross Abstract: While language reasoning models excel in many tasks, visual reasoning remains challenging for current large multimodal models (LMMs).

By Andr\'e G. Viveiros, Nuno Gon\c{c}alves, Matthias Lindemann, Andr\'e Martins

llms rag diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 28

Learning Sampling Parameters for Diffusion Models

arXiv:2607. 23488v1 Announce Type: new Abstract: Text-to-image diffusion models expose many inference-time sampling parameters, including prompts, negative prompts, classifier-free guidance scales, and noise schedules.

By Arisrei Lim, Yossi Gandelsman

llms diffusion reinforcement-learning multimodal

arXiv AI

Jul 28

scMIR: a vision-language foundation model for single-cell light microscopy image representation

arXiv:2607. 22712v1 Announce Type: cross Abstract: Single-cell light microscopy images have become an important data source for characterizing cell phenotypes, but their complexity and heterogeneity pose challenges to high-throughput automated analysis.

By Yifan Shang (Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China, College of Computer Science and Electronic Engineering, Hunan University, Changsha, China), Jiahui Tan (College of Computer Science and Electronic Engineering, Hunan University, Changsha, China), Xiangxiang Zeng (College of Computer Science and Electronic Engineering, Hunan University, Changsha, China), Renjie Zhou (Department of Biomedical Engineering, The Chinese University of Hong Kong, Hong Kong, China)

fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Jul 28

KANEx: Translating Kolmogorov-Arnold Networks' Interpretability to Medical Explainability

arXiv:2607. 24730v1 Announce Type: cross Abstract: Computer vision models have become highly effective for medical applications, yet their black-box nature continues to undermine clinician trust.

By Krithi Shailya, Ananya Lakshmi Ravi, Venkatanathan K. V., Sowmya S. Sundaram, Gokul S. Krishnan, Aditi Anand, Balaraman Ravindran

llms computer-vision multimodal safety

arXiv Machine Learning

Jul 28

WorldDiT: A Unified Diffusion Architecture for World and Action Modeling

arXiv:2607. 23909v1 Announce Type: new Abstract: Many recent robot policies pursue stronger control by using large pretrained vision-language models (VLMs) as the action backbone.

By Sen Wang, R. Gnana Praveen, Bidhan Roy, Marcos Villagra

llms diffusion robotics multimodal

arXiv AI

Jul 28

DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

arXiv:2607. 24516v1 Announce Type: cross Abstract: While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely heuristic: practitioners stack datasets that pass quality filters, set cross-domain ratios by intuition, and lack a principled, attributable criterion for admitting new data, while frontier recipes remain undisclosed.

By Jiahao Xie, Zhongbin Guo, Qianle Wang, Ruiqi Lu, Dongling Xiao, Wanxuan Sun, Cheng Yang

llms multimodal

arXiv Machine Learning

Jul 28

On-Device Inference versus Wireless Streaming: Energy-Efficient Multi-Modal Deep Learning for Wearable Cardiovascular Patches

arXiv:2510. 18668v4 Announce Type: replace Abstract: Wearable cardiovascular sensor patches promise continuous, unobtrusive monitoring, but their tight energy, memory, and compute budgets make it unclear whether physiological signals should be analyzed on the device or streamed to the cloud for processing.

By Mustafa Fuad Rifet Ibrahim, Tunc Alkanat, Felix Manthey, Maurice Meijer, Alexander Schlaefer, Peer Stelldinger

multimodal benchmarks

arXiv AI

Jul 28

DSCH-Loss: A Dynamic Semantic Channel Objective for Deep Semantic Hashing

arXiv:2607. 24567v1 Announce Type: new Abstract: Semantic hashing methods for generating short binary hash codes that allow efficient approximate nearest neighbor search in high-dimensional data spaces have gained extensive consideration in recent years.

By Tobias J. Bauer, Christian Riess, Daniel Loebenberger, Christian Bergler

multimodal benchmarks

arXiv Machine Learning

Jul 28

Analyzing the Importance of Blank for CTC-Based Knowledge Distillation

arXiv:2506. 01503v2 Announce Type: replace Abstract: With the rise of large pre-trained foundation models for automatic speech recognition new challenges appear.

By Benedikt Hilmes, Nick Rossenbach, Ralf Schl\"uter

efficiency multimodal

arXiv AI

Jul 28

DreamCAD: Scaling Multi-modal CAD Generation using Differentiable Parametric Surfaces

arXiv:2603. 05607v2 Announce Type: replace-cross Abstract: Computer-Aided Design (CAD) relies on structured and editable geometric representations, yet existing generative methods are constrained by small annotated datasets with explicit design histories or boundary representation (BRep) labels.

By Mohammad Sadil Khan, Muhammad Usama, Rolandos Alexandros Potamias, Didier Stricker, Muhammad Zeshan Afzal, Jiankang Deng, Ismail Elezi

llms multimodal benchmarks

arXiv AI

Jul 28

Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization

arXiv:2607. 24354v1 Announce Type: new Abstract: Automatic prompt optimization (APO) has been widely adopted to adapt vision-language models (VLMs) to downstream tasks without weight updates, yielding promising results.

By Haoyue Liu, Xiaoyu Ma, Ye Chen, Yuexian Zou, Xiaoying Tang

llms multimodal benchmarks

arXiv AI

Jul 28

AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use

arXiv:2505. 12650v2 Announce Type: replace-cross Abstract: Reconstructing atomistic crystal structures from a single noisy STEM projection is an ill-posed inverse problem: multiple lattices can explain similar contrast, and purely feed-forward models cannot verify physical validity.

By Yaotian Yang, Yiwen Tang, Yizhe Chen, Xiao Chen, Jiangjie Qiu, Hao Xiong, Haoyu Yin, Zhiyao Luo, Yifei Zhang, Sijia Tao, Wentao Li, Qinghua Zhang, Yuqiang Li, Wanli Ouyang, Bin Zhao, Xiaonan Wang, Fei Wei

agents multimodal benchmarks

arXiv Machine Learning

Jul 28

When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents

arXiv:2607. 24077v1 Announce Type: cross Abstract: Optical Character Recognition (OCR) is a key component in the digitization of historical archives.

By Marina Gardella (CB), Camilo Mari{\~n}o (UDELAR, CB), Diego Belzarena (UDELAR, CB), Ignacio Ram{\'i}rez (UDELAR), Gregory Randall (UDELAR), Jean-Michel Morel (LU - Hong Kong)

llms computer-vision multimodal benchmarks safety

arXiv AI

Jul 28

A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference

arXiv:2607. 24148v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have demonstrated strong potential for embodied AI, yet their high inference latency on GPUs limits real-time deployment.

By Zhuoran Song, Haozhe Jiang, Chunyu Qi, Minnan Pei, Gang Li, Xiaoyao Liang, Haibing Guan

robotics efficiency multimodal

arXiv AI

Jul 28

Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training

arXiv:2512. 24125v3 Announce Type: replace-cross Abstract: General-purpose robotic systems operating in open-world environments must achieve both broad generalization and high-precision action execution, a combination that remains challenging for existing Vision-Language-Action (VLA) models.

By Yi Liu, Sukai Wang, Dafeng Wei, Xiaowei Cai, Linqing Zhong, Jiange Yang, Guanghui Ren, Jinyu Zhang, Maoqing Yao, Chuankang Li, Xindong He, Liliang Chen, Jianlan Luo

llms nlp robotics multimodal benchmarks

arXiv AI

Jul 28

From Pixels to Prompts: Vision-Language Models

arXiv:2605. 07544v3 Announce Type: replace Abstract: When you read a paper about a new Vision-Language Model today, it can be easy to forget how strange this idea would have sounded not so long ago.

By Khang Nhat Hoang Vo

llms multimodal benchmarks

Get the daily briefing