Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,972 stories · RSS feed

arXiv AI

Jun 29

Hippocampus-DETR: An Explicit Memory Object Detection Framework Based on Hippocampus Modeling

arXiv:2606. 27831v1 Announce Type: cross Abstract: This paper addresses the lack of explicit memory mechanisms in current object detection models and proposes Hippocampus-DETR, a novel detection framework based on biological hippocampal memory modeling.

By Zhaoning Shi, Bo Ma, Hao Xu, Zepeng Yang, Bo Liang

computer-vision multimodal safety

arXiv Machine Learning

Jun 29

A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

arXiv:2606. 27886v1 Announce Type: new Abstract: Recent advances in Human Activity Recognition (HAR) from wearable sensors have shown that multi-modal deep learning models consistently outperform their uni-modal counterparts.

By Ahmed Mohamady, Robin Burchard, Kristof Van Laerhoven

multimodal benchmarks

arXiv Machine Learning

Jun 29

EchoSonar-R: A Multi-View Reasoning-Enabled Model for Disease Classification and Report Generation in Echocardiography

arXiv:2606. 28164v1 Announce Type: cross Abstract: Echocardiography is the most widely used non-invasive cardiac imaging modality, providing essential information for cardiovascular diagnosis.

By Darya Taratynova, Ahmed Aly, Numan Saeed, Mohammad Yaqub

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 29

Reflect-R1: Evidence-Driven Reflection for Self-Correction in Long Video Understanding

arXiv:2606. 27922v1 Announce Type: cross Abstract: Current multimodal reflection mechanisms for long video understanding predominantly rely on closed-loop self-reflection within internal parameters.

By Shuimu Chen, Yuteng Chen, Yuanshen Guan, Zebang Cheng, Zeyu Zhang, Shengqian Qin, Bin Xia, Jiaran Li, Wenming Yang, Fei Ma

reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 29

What Was That Again? Certified Robustness for Automatic Speech Recognition

arXiv:2606. 27698v1 Announce Type: cross Abstract: Automatic Speech Recognition systems are notoriously both sensitive to adversarial and benign perturbations.

By Andrew C. Cullen, Neil Marchant, Jiani Xie, Paul Montague, Benjamin I. P. Rubinstein

multimodal safety

arXiv AI

Jun 29

Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting

arXiv:2602. 01588v3 Announce Type: replace-cross Abstract: Multimodal time series forecasting is crucial in real-world applications, where decisions depend on both numerical data and contextual signals.

By Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen, Hung Le

rag multimodal benchmarks safety

arXiv Machine Learning

Jun 29

Dual-Learning based Penalized Multi-Align Clustering for Multi-View Incomplete and Disorderly Data

arXiv:2606. 27984v1 Announce Type: new Abstract: Multimodal feature fusion can effectively capture complex patterns in real-world data by integrating complementary information from different modalities.

By Liang Zhao, Shubin Ma, Bo Xu, Qingchen Zhang

multimodal safety

arXiv AI

Jun 29

From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales

arXiv:2604. 08591v2 Announce Type: replace-cross Abstract: Hallucinations in large ASR models present a critical safety risk.

By Ivan Viakhirev, Kirill Borodin, Grach Mkrtchian

multimodal safety

arXiv AI

Jun 29

Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout

arXiv:2605. 05092v2 Announce Type: replace-cross Abstract: Safe L2/L3 driving automation requires anticipating human-in-the-loop reactions during shared-control transitions.

By Haozhuang Chi, Daosheng Qiu, Hao Su, Haochen Liu, Zirui Li, Haoruo Zhang, Chen Lv

diffusion multimodal safety

arXiv AI

Jun 29

Yuvion VL: A Multimodal Foundation Model for Adversarial Content and AI Safety

arXiv:2606. 25034v2 Announce Type: replace-cross Abstract: General-purpose models often struggle to reliably identify and understand real-world multimodal risks, largely due to the inherent multimodal adversarial nature of content and AI safety.

By Shikai Qiu, Xiaowen Xu, Benlei Cui, Ting Ma, Xiufeng Huang, Wenjing Jiang, Shaoxuan He, Haolei Xu, Chunyang Chai, Yujian Li, Yiliang Zhang, Guanghui Wang, Ziheng Wang, Ziwen Xu, Zhaoyu Fan, Jinhao Chen, Ruijie Jian, Hongxing Li, Chuxi Xiao, Xinyue Chen, Wenxuan Liu, Libin Dong, Yupeng Cao, Xiaoqian Xia, Jing Wang, Zhe Jiang, Zhenan Ye, Guang Yang, Bin Liu, Wei Peng, Ziqiang Zhu, Meihui Lian, Kaiwen Lv Kacuila, Haidong Ding, Dongjie Zhang, Yangfan Zhou, Bingyu Zhu, Yan Wang, Hai Zhao, Xuan Jin, Wei Zhao, Pengfei Sun, Huiming Zhang, Wei Wang, Xipeng Cao, Jialun Chen, Xiao Chen, Shaola Ren, Yunqing Hu, Bin Li, Chengwen Yao, Meng Huang, Xianfeng Li, Bin Tang, Chao Liu, Hui Xue, Longtao Huang, Haiwen Hong

llms fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 29

Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training

arXiv:2606. 28104v1 Announce Type: cross Abstract: Vision-based assessment can provide convenient and cost-effective evaluation in Traditional Chinese Medicine (TCM) rehabilitation training, where action quality assessment (AQA) from computer vision offers a promising solution.

By Francis Xiatian Zhang, Hao Yao, Shengxuan Chen, Hong Zhu, Hongxiao Jia, Sisi Zheng, Hubert P. H. Shum

computer-vision robotics multimodal

arXiv AI

Jun 29

MPFlow: Multi-modal Posterior-Guided Flow Matching for Zero-Shot MRI Reconstruction

arXiv:2603. 03710v3 Announce Type: replace-cross Abstract: Zero-shot MRI reconstruction relies on generative priors, but single-modality unconditional priors produce hallucinations under severe ill-posedness.

By Seunghoi Kim, Chen Jin, Henry F. J. Tregidgo, Matteo Figini, Daniel C. Alexander

diffusion computer-vision multimodal safety

arXiv AI

Jun 29

Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding

arXiv:2606. 27596v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) exhibit sophisticated reasoning but remain susceptible to object hallucination.

By Liu Yu, Can Chen, Ping Kuang, Zhikun Feng, Fan Zhou, Gillian Dobbie

llms multimodal benchmarks safety

arXiv AI

Jun 29

Benchmarking Multi-Modal Graph-based Social Media Popularity Prediction

arXiv:2606. 27539v1 Announce Type: cross Abstract: Social media popularity prediction aims to forecast the future reach or influence of online content from early-stage observations.

By Utkarsh Sahu, Zhisheng Qi, Li Zhu, Yizhao Yang, Jun Li, Ryan Rossi, Yu Wang

llms agents multimodal benchmarks

arXiv AI

Jun 29

Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge

arXiv:2606. 27527v1 Announce Type: cross Abstract: Large Language Models (LLMs) possess broad conceptual knowledge acquired through large-scale text pretraining, yet their potential to supervise models in other modalities remains underexplored.

By Thomas Shih-Chao Liang, Zhuoran Yu, Yong Jae Lee

llms efficiency multimodal benchmarks

arXiv AI

Jun 29

MER-R1: Multimodal Emotion Reasoning via Slow-Fast Thinking Synergy

arXiv:2606. 27652v1 Announce Type: new Abstract: We find that explicit reasoning does not necessarily translate into better multimodal emotion recognition (MER) accuracy, even though it makes predictions more interpretable.

By Zhiyuan Han, Beier Zhu, Wenwen Tong, Chengwei Qin, Xinyi Wang, Jiayu Zhang, Jiangnan Chen, Hewei Guo, Dongchuan Ran, Lewei Lu, Xun Yang

reinforcement-learning multimodal benchmarks

arXiv AI

Jun 29

OverFlowLight: Real-Time Gridlock Prevention and Traffic Signal Optimization for Urban Intersections

arXiv:2606. 27381v1 Announce Type: cross Abstract: Queue overflow, a severe consequence of urban traffic congestion, occurs when vehicle queues exceed intersection capacity, obstructing upstream traffic and triggering cascading gridlocks.

By Mingyuan Li, Boyang Huang, Tianqi Jiang, Chenpu Li, Chunyu Liu, Yang Li, Ruimin Li, Qiang Wu

agents reinforcement-learning efficiency multimodal safety

arXiv Machine Learning

Jun 29

Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings

arXiv:2606. 27543v1 Announce Type: cross Abstract: The variations in vocal effort range (e.

By Zahra Omidi, John H. L. Hansen

multimodal benchmarks

arXiv AI

Jun 29

ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

arXiv:2606. 27974v1 Announce Type: cross Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge.

By ZhengXian Wu, Hangrui Xu, Kai Shi, Zhuohong Chen, Yunyao Yu, Chuanrui Zhang, Zirui Liao, Jun Yang, Zhenyu Yang, Haonan Lu, Haoqian Wang

rag agents nlp reinforcement-learning fine-tuning multimodal

arXiv AI

Jun 29

NormAct: A Benchmark for Hidden Social Norm Compliance in Embodied Planning

arXiv:2606. 27826v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly deployed as embodied planners in egocentric environments, where task success requires not only achieving instructed goals but also acting in socially appropriate ways.

By Shiyun Zhao, Xinwei Song, Tianyu Guo, Xiaomeng Gao, Mingyuan Liu, Xu Han, Yuanyuan Zhang, Zhenliang Zhang, Xue Feng, Bo Dai

llms agents robotics multimodal benchmarks

Get the daily briefing