Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,954 stories · RSS feed

arXiv AI

Jul 20

Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

arXiv:2607. 15565v1 Announce Type: cross Abstract: Where should the question go in a vision-language model (VLM) prompt: before the image or after it?

By Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan, Gautam Rajendrakumar Gare

llms nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 20

Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents

arXiv:2607. 15657v1 Announce Type: cross Abstract: Multimodal AI agents increasingly rely on persistent long-term memory to ground generation in past visual and textual episodes.

By Halima Bouzidi, Mboutidem Ekemini Mkpong, Mohammad Abdullah Al Faruque

llms agents multimodal safety

arXiv AI

Jul 20

Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

arXiv:2602. 24009v4 Announce Type: replace-cross Abstract: Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols.

By Zhicheng Fang, Jingjie Zheng, Chenxu Fu, Wei Xu

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jul 20

Knowledge-Guided Cross-Modal Fusion for Adult-to-Pediatric ECG Transfer via Label-Conditioned Contrastive Alignment

arXiv:2607. 15928v1 Announce Type: new Abstract: Adult and pediatric electrocardiogram (ECG) interpretation relies on age-sensitive criteria, and models pretrained mainly on adult ECGs often transfer poorly to pediatric populations when pediatric labels are scarce.

By Xinran Liu, Yuwen Li, Hongxiang Gao, Heyang Xu, Jianqing Li, Zongmin Wang, Chengyu Liu

rag fine-tuning multimodal safety

arXiv Machine Learning

Jul 20

Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation

arXiv:2607. 15880v1 Announce Type: cross Abstract: Imitation Learning aims to learn skills from extensive observations and demonstrations for robots, so it suffers from data scarcity and environment generalization.

By Zhenduo Shang, Xiyao Liu, Bohan Li, Xudong Wang, Teng Ren, Lianqing Liu, Zhi Han

agents diffusion robotics fine-tuning multimodal benchmarks

arXiv AI

Jul 20

Map as a Prompt: Learning Multi-Modal Spatial-Signal Foundation Models for Cross-scenario Wireless Localization

arXiv:2607. 15713v1 Announce Type: cross Abstract: Accurate and robust wireless localization is a critical enabler for emerging 5G/6G applications, including autonomous driving, extended reality, and smart manufacturing.

By Yong Chu, Xun Zhou, Zenglin Xu, Hui Wang, Yue Yu

agents robotics multimodal benchmarks

arXiv AI

Jul 20

DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings

arXiv:2607. 15418v1 Announce Type: new Abstract: We introduce DrawingVQA, the first benchmark designed to evaluate multimodal large language models (MLLMs) on real-world construction drawings -- a core media in architecture, civil, and many other engineering practices.

By Yoonhwa Jung, Junryu Fu, Mani Golparvar-Fard

llms multimodal benchmarks

arXiv AI

Jul 20

FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment

arXiv:2604. 23786v2 Announce Type: replace Abstract: In recent years, the integration of multimodal machine learning in wellbeing assessment has offered transformative potential for monitoring mental health.

By Sophie Chiang, Tom Brennan, Fethiye Irmak Dogan, Jiaee Cheong, Hatice Gunes

llms multimodal safety

arXiv AI

Jul 20

IMBench: A Benchmark for Intuitive Robotic Manipulation

arXiv:2607. 15641v1 Announce Type: cross Abstract: Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints.

By Anurag Maurya, Sukhvansh Jain, Prajwal Avhad, Gautham Balachandran, Ziyi Zhou, Atharva Kshirsagar, Satyam Singh, Bowen Li. Rishabh Mukund, Ritul Singh, Jatin Vira, Suvonil Chatterjee, Devesh K. Jha

llms agents robotics multimodal benchmarks

arXiv Machine Learning

Jul 20

RobustSpeechFlow: Learning Robust Text-to-Speech Trajectories via Augmentation-based Contrastive Flow Matching

arXiv:2605. 22083v2 Announce Type: replace-cross Abstract: While flow-matching text-to-speech (TTS) achieves strong zero-shot speaker similarity and naturalness, it remains susceptible to content fidelity issues, particularly skip and repeat errors from imperfect alignment.

By Jinhyeok Yang, Hyeongju Kim, Yechan Yu, Joon Byun, Frederik Bous, Juheon Lee

diffusion multimodal benchmarks safety

arXiv AI

Jul 20

EgoExoMoCap: Distributed Ego-Exo Human Motion Capture

arXiv:2607. 15868v1 Announce Type: cross Abstract: Human motion capture from head-mounted devices (HMDs) offers a scalable way to acquire real-world human motion and interaction data, which is crucial for applications in embodied AI and VR/AR.

By Jiaxi Jiang, Bharat Lal Bhatnagar, Nan Yang, Lingni Ma, Sebastian Starke, Robin Kips, Nadine Bertsch, Christian Holz, Federica Bogo

robotics multimodal

arXiv AI

Jul 20

LVSum: A Benchmark for Timestamp-Aware Long Video Summarization

arXiv:2604. 10024v2 Announce Type: replace-cross Abstract: Long video summarization presents significant challenges for multimodal large language models (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded.

By Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng, Ganesh Nagarajan

llms nlp multimodal benchmarks safety

arXiv AI

Jul 20

Perception-Aligned AI Outputs: End-to-End Visual Prediction for Uncertainty Communication in Clinical Decision-Making

arXiv:2205. 04599v2 Announce Type: replace-cross Abstract: Explainable Artificial Intelligence (XAI) is essential for trustworthy AI in healthcare, yet many existing methods rely on technical explanations that are difficult for clinicians and patients to interpret.

By Mohammad Eslami, Solale Tabarestani, Saber Kazeminasab, Ehsan Adeli, Glyn Elwyn, Tobias Elze, Mengyu Wang, Nazlee Zebardast, Lucia Sobrin, Nassir Navab, Daniel Shu Wei Ting, Malek Adjouadi

multimodal safety

arXiv AI

Jul 20

SLAPBench: Benchmarking Multimodal Large Language Models for Four-Finger SLAP Fingerprint Verification

arXiv:2607. 15517v1 Announce Type: cross Abstract: Four-finger SLAP fingerprints are flat live-scan impressions of the index, middle, ring, and little fingers of one hand, used for identity verification in border control and law enforcement.

By Bibesh Pyakurel, M. G. Sarwar Murshed

llms multimodal benchmarks safety

arXiv AI

Jul 20

An Exam for Active Observers

arXiv:2607. 16165v1 Announce Type: cross Abstract: Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot.

By Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger

llms multimodal benchmarks

arXiv Machine Learning

Jul 20

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

arXiv:2607. 15942v1 Announce Type: cross Abstract: Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks.

By Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"), Luc Van Gool (INSAIT, Sofia University "St. Kliment Ohridski"), Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")

llms computer-vision reinforcement-learning multimodal benchmarks safety

arXiv AI

Jul 20

MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

arXiv:2607. 15592v1 Announce Type: new Abstract: Multimodal Knowledge Graph Completion (MKGC) requires inferring missing entities from structural, textual, and visual cues.

By Xu Hou, Meiyu Liang, Wei Huang, Yawen Li, Zhe Xue, Wu Liu, Guanhua Ye, Lei Shi, Kangkang Lu

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 20

Natural Backdoor Attacks on Speech Recognition Models

arXiv:2607. 15724v1 Announce Type: cross Abstract: With the rapid development of deep learning, its vulnerability has gradually emerged in recent years.

By Jinwen Xin, Xixiang Lyu, Jing Ma

multimodal

arXiv AI

Jul 20

On the Geometry of Learned Representations in Event-Based Multi-Modal Egomotion Estimation

arXiv:2607. 15794v1 Announce Type: cross Abstract: Classical approaches to event-based egomotion estimation, including those adopted by the top-performing teams of the ELOPE challenge, rely on geometric optimization frameworks such as contrast maximization, homography estimation, or dense optical flow combined with analytic motion inversion.

By Stefano Silvestrini, Michele Ceresoli

rag diffusion computer-vision multimodal

arXiv Machine Learning

Jul 20

Diffusion models recover accurate mixture weights despite score function insensitivity

arXiv:2607. 15485v1 Announce Type: new Abstract: Score-based generative models exhibit a puzzling behavior: they often appear to cover all modes of a target multimodal distribution and yet may fail to learn the correct relative mode amplitudes, which can be interpreted as mixture weights.

By Andrew Dennehy, Ramchandran Muthukumar, Rebecca Willett, Nisha Chandramoorthy

diffusion multimodal benchmarks

Get the daily briefing