Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,414 stories · RSS feed

arXiv AI

Jun 4

From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models

arXiv:2512. 05277v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances.

By Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari

llms agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

arXiv:2606. 04434v1 Announce Type: cross Abstract: Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks.

By Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh Afghah

llms efficiency multimodal benchmarks

arXiv AI

Jun 4

Early Detection of Alzheimer's Disease Using Explainable Machine Learning on Clinical Biomarkers: A Multi-Class Classification Study Using the Alzheimer's Disease Neuroimaging Initiative (ADNI) Dataset

arXiv:2606. 03995v1 Announce Type: cross Abstract: Background: Alzheimer's disease (AD) affects over 55 million people worldwide.

By Afshan Hashmi

multimodal

arXiv Machine Learning

Jun 4

A Geometric View of Counterfactual Behavior: Interaction of Boundary Proximity and Local Support

arXiv:2606. 04209v1 Announce Type: new Abstract: Counterfactual explanations seek small, semantically meaningful changes to an input that alter a model's prediction, and are widely used to interpret and audit machine learning systems.

By Ioanna Gemou, Matteo Gamba, Randall Balestriero, Ritambhara Singh

multimodal

arXiv AI

Jun 4

MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models

arXiv:2606. 04349v1 Announce Type: cross Abstract: Conventional Post-Training Quantization (PTQ) methods struggle with 4-bit Omni-modal Large Language Models (OLLMs) due to the extreme distribution heterogeneity and disparate outlier patterns across modalities.

By Yue Wu, Changyuan Wang, Zixuan Wang, Shilin Ma, Yansong Tang

llms efficiency multimodal benchmarks safety

arXiv AI

Jun 4

Can Generalist Agents Automate Data Curation?

arXiv:2606. 04261v1 Announce Type: new Abstract: Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback.

By Feiyang Kang, Hanze Li, Adam Nguyen, Mahavir Dabas, Jiaqi W. Ma, Frederic Sala, Dawn Song, Ruoxi Jia

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 4

LMM-IR: Large-Scale Netlist-Aware Multimodal Framework for Static IR-Drop Prediction

arXiv:2511. 12581v2 Announce Type: replace Abstract: Static IR drop analysis is a fundamental and critical task in the field of chip design.

By Kai Ma, Zhen Wang, Hongquan He, Qi Xu, Tinghuan Chen, Hao Geng

llms diffusion multimodal benchmarks

arXiv AI

Jun 4

Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?

arXiv:2506. 10912v4 Announce Type: replace Abstract: Toxicity remains a leading cause of early-stage drug development failure.

By Fei Lin, Ziyang Gong, Cong Wang, Tengchao Zhang, Yonglin Tian, Yining Jiang, Ji Dai, Chao Guo, Xiaotong Yu, Xue Yang, Gen Luo, Fei-Yue Wang

llms efficiency multimodal benchmarks safety

arXiv AI

Jun 4

Belief-Aware VLM Model for Human-like Reasoning

arXiv:2604. 09686v2 Announce Type: replace Abstract: Traditional neural network models for intent inference rely heavily on observable states and struggle to generalize across diverse tasks and dynamic environments.

By Anshul Nayak, Shahil Shaik, Yue Wang

llms diffusion reinforcement-learning multimodal

arXiv AI

Jun 4

Building The Ph(ysical)AI Layer Of Machine Intelligence

arXiv:2606. 04106v1 Announce Type: cross Abstract: Foundation models achieve generalization through massive-scale training on diverse data, but have limitations with transfer to truly unseen domains without paired training data.

By Ulbert Jose Botero, Liam Smith, Brooks Olney, Pooya Khorrami, Steven Kusiak, Watson Jia, Sage Trudeau, Daniel Capecci

fine-tuning multimodal

arXiv AI

Jun 4

Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching

arXiv:2512. 03553v3 Announce Type: replace-cross Abstract: Content moderation remains a critical yet challenging task for large-scale user-generated video platforms, especially in livestreaming environments where moderation must be timely, multimodal, and robust to evolving forms of unwanted content.

By Wei Chee Yew, Hailun Xu, Sanjay Saha, Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Kanchan Sarkar, Zhenheng Yang, Danhui Guan

llms multimodal safety

arXiv AI

Jun 4

GenSpan: Generation-Calibrated Motion Span Priors for Multi-Verb Video Corpus Moment Retrieval

arXiv:2603. 22121v2 Announce Type: replace-cross Abstract: Video Corpus Moment Retrieval (VCMR) aims to retrieve both the correct video and its temporal segment corresponding to a natural-language query, a task that is especially challenging for multi-verb queries where temporal action ordering is critical.

By Yunzhuo Sun, Xinyue Liu, Yanyang Li, Nanding Wu, Linlin Zong, Xianchao Zhang, Wenxin Liang

llms multimodal benchmarks

arXiv AI

Jun 4

DetectZoo: A Unified Toolkit for AI-Generated Content Detection Across Text, Audio, and Image Modalities

arXiv:2606. 04205v1 Announce Type: cross Abstract: The growing popularity and capacity of generative models have eroded the distinction between human and machine-generated content, motivating a growing body of work on detection across text, images, and audio.

By Sajad Ebrahimi, Nima Jamali, Bardia Shirsalimian, Kelly McConvey, Wentao Zhang, Jalehsadat Mahdavimoghaddam, Maksym Taranukhin, Maura Grossman, Vered Shwartz, Yuntian Deng, Ebrahim Bagheri

multimodal benchmarks

arXiv AI

Jun 4

DEFLECT: Temporal Counterfactual Preference Learning for Delay-Robust Asynchronous VLAs

arXiv:2605. 19294v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) policies increasingly rely on asynchronous inference to hide large-model latency behind ongoing robot motion.

By Yixiang Zhu, Yonghao Chen, Zijie Yang, Yusong Hu, Xinyu Chen

robotics fine-tuning multimodal safety

arXiv AI

Jun 4

Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance

arXiv:2606. 04970v1 Announce Type: cross Abstract: We envision a proactive multi-modal assistant system which gives users real-time step-by-step guidance on a procedural task, autonomously deciding \textit{when} to interrupt, and \textit{how} to coach.

By Kaustav Kundu, Ritvik Shrivastava, Maxim Arap, Nanshu Wang, Xianhui Zhu, Quintin Fettes, Gautam Tiwari, Parth Suresh, Th\'eo Moutakanni, Alejandro Castillejo Munoz, Allen Bolourchi, Pascale Fung, Pinar Donmez, Babak Damavandi, Anuj Kumar, Seungwhan Moon

llms multimodal benchmarks

arXiv AI

Jun 4

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models

arXiv:2606. 04381v1 Announce Type: cross Abstract: Recent large language models (LLMs) often appear to exhibit spatial reasoning ability; however, this capability is largely \emph{symbolic}, arising from pattern matching over spatial language rather than true \emph{geometric} reasoning over space.

By Chen Chu, Bita Azarijoo, Li Xiong, Khurram Shafique, Cyrus Shahabi

llms multimodal benchmarks

arXiv AI

Jun 4

Continual Visual and Verbal Learning Through a Child's Egocentric Input

arXiv:2606. 05115v1 Announce Type: cross Abstract: Children learn the meanings of words from a continuous, temporally structured stream of egocentric experience.

By Xiaoyang Jiang, Yanlai Yang, Kenneth A. Norman, Brenden Lake, Mengye Ren

computer-vision multimodal benchmarks

arXiv AI

Jun 4

LaVIDE: Language-Prompted Satellite Change Detection via Map-Image Alignment

arXiv:2411. 19758v2 Announce Type: replace-cross Abstract: Remote sensing change detection based on a map reference and an up-to-date image boosts timely observation of the Earth's surface when earlier images are lacking for comparison.

By Shuguo Jiang, Fang Xu, Chuandong Liu, Hong Tan, Shengyang Li, Lei Yu, Wen Yang, Sen Jia, Gui-Song Xia

rag computer-vision multimodal benchmarks safety

arXiv AI

Jun 4

Revisiting Model Stitching In the Foundation Model Era

arXiv:2603. 12433v3 Announce Type: replace-cross Abstract: Model stitching, connecting early layers of one model (source) to later layers of another (target) via a light stitch layer, has served as a probe of representational compatibility.

By Zheda Mai, Ke Zhang, Fu-En Wang, Zixiao Ken Wang, Albert Y. C. Chen, Lu Xia, Min Sun, Wei-Lun Chao, Cheng-Hao Kuo

llms multimodal

arXiv Machine Learning

Jun 4

RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities

arXiv:2606. 05109v1 Announce Type: new Abstract: To leverage the full potential of multimodal data, we need representations that go beyond the state-of-the-art alignment and fusion approaches and exploit all cross-modal interactions without sacrificing modality-specific information.

By Vasiliki Rizou, Pascal Frossard, Dorina Thanou

rag multimodal benchmarks safety

Get the daily briefing