Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,869 stories · RSS feed

arXiv AI

Jul 28

Chart Deception in Vision-Language Models: From Vulnerability to Mitigation

arXiv:2607. 22600v1 Announce Type: new Abstract: Information visualizations are widely used to communicate patterns, trends, and outliers, yet deceptive design choices-such as truncated or inverted axes, distorted aspect ratios, inappropriate encodings, and misleading color mappings-can systematically alter interpretation while preserving the underlying data.

By Ridwan Mahbub, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Mizanur Rahman, Mir Tafseer Nayeem, Enamul Hoque

llms agents multimodal benchmarks

arXiv AI

Jul 28

DSCH-Loss: A Dynamic Semantic Channel Objective for Deep Semantic Hashing

arXiv:2607. 24567v1 Announce Type: new Abstract: Semantic hashing methods for generating short binary hash codes that allow efficient approximate nearest neighbor search in high-dimensional data spaces have gained extensive consideration in recent years.

By Tobias J. Bauer, Christian Riess, Daniel Loebenberger, Christian Bergler

multimodal benchmarks

arXiv AI

Jul 28

Cheap Probes Predict Expensive Training in 3D-CT Vision--Language Models

arXiv:2607. 22771v1 Announce Type: cross Abstract: Picking the frozen image encoder for a 3D~CT vision--language model (VLM), together with the token-compression scheme on top of it, is a search over many candidates.

By Renjie Liang

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jul 28

Explaining BiomedCLIP with Weighted Banzhaf Interactions Supported by Tree-Gram Parsing

arXiv:2607. 23368v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are demonstrating significant capabilities in medical tasks like radiology analysis, yet providing faithful and interpretable explanations remains a key consideration for their responsible deployment in clinical settings.

By Jakub Rymarski (University of Warsaw, Poland), Adam Rempa{\l}a (University of Warsaw, Poland), Bart{\l}omiej Sobieski (University of Warsaw, Poland), Przemys{\l}aw Biecek (University of Warsaw, Poland)

llms nlp multimodal

arXiv Machine Learning

Jul 28

To Erase, or Not to Erase: Robust Training-Free Concept Erasure with Preservation aware Adaptive Ranked Subspace Expansion

arXiv:2607. 23492v1 Announce Type: cross Abstract: Concept erasure techniques (CETs) edit text-to-image diffusion models to erase undesired targets such as NSFW content or copyrighted styles, while preserving model utility on benign concepts.

By Shaswati Saha, Rajasekhar Anguluri, Manas Gaur

llms diffusion multimodal

arXiv AI

Jul 28

Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training

arXiv:2512. 24125v3 Announce Type: replace-cross Abstract: General-purpose robotic systems operating in open-world environments must achieve both broad generalization and high-precision action execution, a combination that remains challenging for existing Vision-Language-Action (VLA) models.

By Yi Liu, Sukai Wang, Dafeng Wei, Xiaowei Cai, Linqing Zhong, Jiange Yang, Guanghui Ren, Jinyu Zhang, Maoqing Yao, Chuankang Li, Xindong He, Liliang Chen, Jianlan Luo

llms nlp robotics multimodal benchmarks

arXiv AI

Jul 28

Like a Baby: Visually Situated Neural Language Acquisition

arXiv:1805. 11546v3 Announce Type: replace-cross Abstract: We examine the benefits of visual context in training neural language models to perform next-word prediction.

By Alexander G. Ororbia, Ankur Mali, Mary Alexandria Kelly, David Reitter

llms rag fine-tuning multimodal benchmarks

arXiv AI

Jul 28

Voice-Driven Semantic Perception for UAV-Assisted Emergency Networks

arXiv:2602. 17394v2 Announce Type: replace-cross Abstract: Unmanned Aerial Vehicle (UAV)-assisted networks are increasingly foreseen as a promising approach for emergency response, providing rapid, flexible, and resilient communications in environments where terrestrial infrastructure is degraded or unavailable.

By Nuno Saavedra, Pedro Ribeiro, Andr\'e Coelho, Rui Campos

llms nlp multimodal

arXiv AI

Jul 28

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

arXiv:2607. 24743v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment.

By Hangjie Yuan, Yichen Qian, Zhiwei Tang, Xianzhe Xu, Lirong Wu, Sicheng Yang, Jinwang Wang, Pengju Wang, Zhitao Zeng, Yizeng Han, Yan Xing, Shengxuan Luo, Tao Feng, Qing Xie, Weigen Yao, Yi Yang, Zuozhu Liu, Jiasheng Tang, Shaocheng Wang, Jitao Wang, Jiahong Dong, Weihua Chen, Feng Xu, Fan Wang

llms rag agents nlp multimodal benchmarks

arXiv Machine Learning

Jul 28

SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

arXiv:2607. 23346v1 Announce Type: new Abstract: Modern deep neural networks are potent catalysts for scientific and industrial impact, yet excessive parameter counts impede deployment in low-compute settings such as hospital equipment and energy infrastructure.

By Aditya Dewan, Arjun Yogeswaran, Benjamin Fedoruk

rag fine-tuning efficiency multimodal benchmarks

arXiv AI

Jul 28

A Few Words Go a Long Way: Language Guided Robot Policy Synthesis

arXiv:2607. 23784v1 Announce Type: cross Abstract: While vision-language-action models have demonstrated impressive zero-shot manipulation capabilities, they remain fundamentally black box policies that are difficult to interpret, adapt, or correct when they inevitably fail.

By Daphne Chen, Archit Ritesh Jain, Eric Goossen, Emma Romig, Michael Murray, Nick Walker, Maya Cakmak

llms agents robotics multimodal benchmarks

arXiv AI

Jul 28

Real2Sim2Real for Vision-Language-Action Manipulation: An AMD ROCm-Based Pipeline

arXiv:2607. 22997v1 Announce Type: cross Abstract: Physical AI -- the integration of large vision-language-action (VLA) models with embodied agents that act in the real world -- has emerged as the next major frontier for AI, echoed by industry leaders such as Jensen Huang (``the next big thing is Physical AI, AI with a body,'' GTC Paris, June 2025) and Dr.

By Qing Yang, Xun Wang, Ziguan Wang, Zhenjiang Li, Hongqiang Wang, Dongdong Weng

agents reinforcement-learning robotics multimodal

arXiv AI

Jul 28

From Pixels to Prompts: Vision-Language Models

arXiv:2605. 07544v3 Announce Type: replace Abstract: When you read a paper about a new Vision-Language Model today, it can be easy to forget how strange this idea would have sounded not so long ago.

By Khang Nhat Hoang Vo

llms multimodal benchmarks

arXiv AI

Jul 28

An Interactive Vision Language Platform for Cognitive Remediation in Schizophrenia

arXiv:2607. 22721v1 Announce Type: cross Abstract: Cognitive remediation tasks often require patients to perform structured actions involving object manipulation and sequential reasoning.

By Nassira Ait Mehdi, Milissa Temmam, Slimane Larabi

llms robotics fine-tuning multimodal

arXiv AI

Jul 28

GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models

arXiv:2607. 23913v1 Announce Type: new Abstract: Modern vision-language models (VLMs) increasingly rely on dynamic or high-resolution visual encoding, producing thousands of visual tokens that substantially increase downstream language-model inference cost.

By Jun Ling, Tao Huang, Junzhuo Liu, Bowen Tang, Peng Wang

llms multimodal benchmarks

arXiv Machine Learning

Jul 28

LanteRn: Latent Visual Structured Reasoning

arXiv:2603. 25629v2 Announce Type: replace-cross Abstract: While language reasoning models excel in many tasks, visual reasoning remains challenging for current large multimodal models (LMMs).

By Andr\'e G. Viveiros, Nuno Gon\c{c}alves, Matthias Lindemann, Andr\'e Martins

llms rag diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jul 28

D3O: Dynamic Distribution Distillation for Ordinal Regression

arXiv:2607. 23575v1 Announce Type: cross Abstract: Ordinal regression is widely used in scenarios where labels are discrete yet inherently ordered.

By Chunlai Dong, Yaojun Hu, Yuyang Xu, Haochao Ying, Jian Wu

efficiency multimodal safety

arXiv AI

Jul 28

MulRobBench: A Decision-Level Benchmark for Safe and Security-Policy-Compliant Multimodal UAV Agents

arXiv:2607. 23870v1 Announce Type: cross Abstract: Smart-city airspace is transforming Uncrewed Aerial Vehicles (UAVs) from passive sensing platforms into cyber-physical decision makers that must follow operational rules under degraded observations and ambiguous language.

By Belal S. Alsinglawi, Weizheng Wang, Junyi Wu, Yi Jiang, Lianhai Lin, Merouane Debbah, Izzat Alsmadi

agents multimodal benchmarks safety

arXiv AI

Jul 28

Controlling Embedding Spaces with Text-Conditioned Transformations

arXiv:2607. 22919v1 Announce Type: cross Abstract: Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification.

By Joseph Fioresi, Fabian Caba Heilbron, Pankaj Nathani, Mubarak Shah, Kushal Kafle

rag diffusion multimodal benchmarks

arXiv AI

Jul 28

A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference

arXiv:2607. 24148v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have demonstrated strong potential for embodied AI, yet their high inference latency on GPUs limits real-time deployment.

By Zhuoran Song, Haozhe Jiang, Chunyu Qi, Minnan Pei, Gang Li, Xiaoyao Liang, Haibing Guan

robotics efficiency multimodal

Get the daily briefing