Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,972 stories · RSS feed

arXiv AI

Jul 7

Responsibility Distribution Estimation in Ego-View Accident Videos with Multimodal Large Language Models

arXiv:2607. 03591v1 Announce Type: cross Abstract: Recent studies on multimodal traffic accident understanding have mainly relied on infrastructure-camera footage, satellite imagery, or structured crash records.

By Ryosei Tamura, Andrew Shin

llms agents computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 7

K9-Bench: Evaluating Multimodal LLMs on Canine-Centric Videos

arXiv:2607. 02680v1 Announce Type: cross Abstract: MLLMs have shown strong zero-shot capabilities across diverse inputs such as across images, video, audio, and text.

By Khush Attarde, Yusuf Ali, Megha Thukral, Divye Bhutani, Thomas Ploetz, Zsolt Kira

llms robotics multimodal benchmarks safety

arXiv AI

Jul 7

Echoes of Unrest: A Multimodal NLP Framework for Early Warning of Fake News and Violence-Driven Mob Activity

arXiv:2607. 02734v1 Announce Type: cross Abstract: Rapid growth in social media has transformed global communication by enabling fast information exchange, but it has also accelerated the spread of misinformation.

By Md. Maruf Bangabashi, Tahmid Hasan, Golam Mahmud, Md. Mostafijur Rahman, Md. Toufiqur Rahman, Jahanur Biswas

rag nlp multimodal benchmarks

arXiv AI

Jul 7

VISOR++: Universal Visual Inputs based Steering for Large Vision Language Models

arXiv:2509. 25533v2 Announce Type: replace-cross Abstract: As Vision Language Models (VLMs) are deployed across safety-critical applications, understanding and controlling their behavioral patterns has become increasingly important.

By Ravikumar Balakrishnan, Mansi Phute

llms multimodal benchmarks safety

arXiv AI

Jul 7

AnchorVLA: Bridging Discrete Decisions and Continuous Trajectories for Vision-Language-Action Planning

arXiv:2607. 03182v1 Announce Type: cross Abstract: Autonomous driving planning requires translating navigation intent, traffic rules, dynamic interactions, and language instructions into executable continuous trajectories.

By Qi Liu, Yabei Li, Hongsong Wang, Heng Zhang, Lei He

llms agents robotics multimodal benchmarks safety

arXiv AI

Jul 7

IRIS: An Intelligent Vision-Language System for Ocular Surface Diseases via Topic Tree and Scene-Driven VQA Generation

arXiv:2607. 04344v1 Announce Type: cross Abstract: While Large Vision-Language Models (VLMs) demonstrate remarkable generic capabilities, their clinical reasoning in specialized domains like ocular surface diseases (OSDs) is severely hindered by a paucity of high-fidelity, multimodal instruction-tuning data.

By Hao Wei, Wenjin Qi, Dasen Dai, Minqing Zhang, Wu Yuan

llms multimodal benchmarks

arXiv AI

Jul 7

Scalable Semantic Steering of Embedding Projections

arXiv:2607. 03978v1 Announce Type: cross Abstract: Low-dimensional projections support interactive visual analysis of high-dimensional data embeddings, but their structure often does not align with analyst-defined semantic relationships.

By Wei Liu, Eric Krokos, Kirsten Whitley, Rebecca Faust, Chris North

llms rag multimodal safety

arXiv AI

Jul 7

MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understanding

arXiv:2607. 02934v1 Announce Type: cross Abstract: Materials phase diagrams are a core knowledge representation in materials science, encoding temperature,composition, phase stability, and phase transformation pathways, with their full understanding requiring thermodynamic mechanism analysis and scientific reasoning.

By Hanwen Wang, Sihan Liang, Zhiwei Liu, Yangang Wang, Wei Yan, Yuqin Liu, Zongguo Wang

multimodal benchmarks safety

arXiv Machine Learning

Jul 7

RL Forgets! Towards Continual Policy Optimization

arXiv:2607. 04364v1 Announce Type: new Abstract: Continual post-training is becoming a central paradigm for adapting vision-language models to evolving tasks.

By Mao-Lin Luo, Zhe-Xu Wang, Zi-Hao Zhou, Bo Ye, Jian Zhao, Min-Ling Zhang, Tong Wei

llms reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jul 7

MoP-JEPA: Hard-Assigned Predictor Mixtures for Stochastic JEPA World Models

arXiv:2607. 05238v1 Announce Type: new Abstract: JEPA world models predict the next latent state with a single deterministic predictor trained by latent regression.

By Zhi Song, Ximing Xing, Zhenchao Tang, hanbo Huang, Tianxu Lv, minghao Yang, Zhongzheng Niu, He Bing, Lusheng Wang, Jianhua Yao

rag multimodal

arXiv AI

Jul 7

Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval

arXiv:2607. 04605v1 Announce Type: cross Abstract: Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive.

By Suhyeong Park, Junha Jung, Jungwoo Park, Jaewoo Kang

efficiency multimodal

arXiv Machine Learning

Jul 7

Pathways of Visual Information Flow in Vision-Language Models

arXiv:2607. 03358v1 Announce Type: cross Abstract: We study how visual information is routed in vision-language models (VLMs).

By Israfel Salazar, Stella Frank, Dan Oneata, Desmond Elliott, Constanza Fierro

llms multimodal benchmarks

arXiv Machine Learning

Jul 7

Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions

arXiv:2604. 11730v4 Announce Type: replace-cross Abstract: Using behavioural science, health interventions focus on behaviour change by providing a framework to help patients acquire and maintain healthy habits that improve medical outcomes.

By Manuela Gonz\'alez-Gonz\'alez, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Lorenzo Sia, Nicolas Richet, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L Bacon, Eric Granger

llms fine-tuning multimodal

arXiv AI

Jul 7

EGRA:Toward Enhanced Behavior Graphs and Representation Alignment for Multimodal Recommendation

arXiv:2508. 16170v2 Announce Type: replace-cross Abstract: MultiModal Recommendation (MMR) systems have emerged as a promising solution for improving recommendation quality by leveraging rich item-side modality information, prompting a surge of diverse methods.

By Xiaoxiong Zhang, Xin Zhou, Zhiwei Zeng, Yongjie Wang, Zhiqi Shen

llms efficiency multimodal safety

arXiv AI

Jul 7

From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model

arXiv:2607. 05396v1 Announce Type: cross Abstract: Real-world robot deployment rarely maintains the training-stage camera setup, where cameras often experience repositioning or remounting depending on actual scenarios.

By Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Shijian Lu, Gongjie Zhang, Ran Xu

robotics multimodal

arXiv Machine Learning

Jul 7

VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wild

arXiv:2410. 00296v2 Announce Type: replace Abstract: Vision-language Models (VLMs) are essential for contextual understanding of both visual and textual information.

By Junlin Fang, Wenyu Chen, Reshmi Ghosh, Robert Sim, Ahmed Salem, Vitor R. Carvalho, Emily Lawton, Sharon Li, Jack W. Stokes, Sean Du

llms multimodal benchmarks

arXiv AI

Jul 7

DSWAM: A Dual-System World Action Foundation Model for Fine-Grained Robot Manipulation

arXiv:2607. 04927v1 Announce Type: cross Abstract: World Action Models (WAMs) provide a promising alternative to Vision-Language-Action (VLA) policies by using video-based world modeling as dense supervision for robot action learning.

By Jian Zhu, Jianjun Zhang, Taiyi Su, Tianbin Liu, Zhangyuan Wang, Kai Xie, Zitai Huang, Chong Ma, Youzhang He, Tianjian Wang, Hanyang Wang, Weihao Ding, Yi Xu

rag diffusion robotics multimodal

arXiv AI

Jul 7

Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoning

arXiv:2607. 04681v1 Announce Type: cross Abstract: Embodied Chain-of-Thought has emerged as a promising mechanism to enhance robot decision-making and interpretability in black-box Vision-Language Action (VLA) models.

By Matthew Foutter, Matteo Cercola, Lena Wild, Yunshan Wang, Michelle Li, Daniele Gammelli, Marco Pavone

agents reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Jul 7

TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models

arXiv:2607. 04593v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated impressive capabilities across different tasks, but their computational cost is dominated by the large number of visual tokens fed to the language model.

By Riccardo Renzulli, Gabriele Spadaro, Shruthi Gowda, Alaa Eddine Mazouz, Van-Tam Nguyen

llms diffusion fine-tuning efficiency multimodal benchmarks

arXiv AI

Jul 7

Hierarchical Multi-to-Single-Modal Knowledge Distillation for Disruption Prediction in EAST

arXiv:2607. 04241v1 Announce Type: cross Abstract: Plasma disruption is a critical threat to tokamak safety.

By Qiang Chen, Xiao Wang, Hao Si, Qingquan Yang, Meiwen Chen, Jianhua Yang, Xiaofeng Han, Yunhu Jia, Ran Chen, Liang Wang, Jin Tang, Guosheng Xu

llms efficiency multimodal safety

Get the daily briefing