Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,498 stories · RSS feed

arXiv AI

Jun 29

Room for Error: Large-Scale Simulation of Over-the-Air Acoustic Attacks

arXiv:2606. 27701v1 Announce Type: cross Abstract: While voice control is rapidly becoming a ubiquitous vector of human-AI communication, the risks facing these systems remain poorly understood.

By Andrew C. Cullen, Neil Marchant, Jiani Xie, Paul Montague, Benjamin I. P. Rubinstein

efficiency multimodal safety

arXiv AI

Jun 29

Yuvion VL: A Multimodal Foundation Model for Adversarial Content and AI Safety

arXiv:2606. 25034v2 Announce Type: replace-cross Abstract: General-purpose models often struggle to reliably identify and understand real-world multimodal risks, largely due to the inherent multimodal adversarial nature of content and AI safety.

By Shikai Qiu, Xiaowen Xu, Benlei Cui, Ting Ma, Xiufeng Huang, Wenjing Jiang, Shaoxuan He, Haolei Xu, Chunyang Chai, Yujian Li, Yiliang Zhang, Guanghui Wang, Ziheng Wang, Ziwen Xu, Zhaoyu Fan, Jinhao Chen, Ruijie Jian, Hongxing Li, Chuxi Xiao, Xinyue Chen, Wenxuan Liu, Libin Dong, Yupeng Cao, Xiaoqian Xia, Jing Wang, Zhe Jiang, Zhenan Ye, Guang Yang, Bin Liu, Wei Peng, Ziqiang Zhu, Meihui Lian, Kaiwen Lv Kacuila, Haidong Ding, Dongjie Zhang, Yangfan Zhou, Bingyu Zhu, Yan Wang, Hai Zhao, Xuan Jin, Wei Zhao, Pengfei Sun, Huiming Zhang, Wei Wang, Xipeng Cao, Jialun Chen, Xiao Chen, Shaola Ren, Yunqing Hu, Bin Li, Chengwen Yao, Meng Huang, Xianfeng Li, Bin Tang, Chao Liu, Hui Xue, Longtao Huang, Haiwen Hong

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 29

HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answering

arXiv:2603. 18558v2 Announce Type: replace-cross Abstract: Long-form video question answering requires reasoning over extended temporal contexts, making frame selection a critical bottleneck for multi-modal large language models (MLLMs) bound by finite context windows.

By Dan Ben-Ami, Gabriele Serussi, Kobi Cohen, Chaim Baskin

llms rag computer-vision nlp multimodal benchmarks

arXiv Machine Learning

Jun 29

A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

arXiv:2606. 27886v1 Announce Type: new Abstract: Recent advances in Human Activity Recognition (HAR) from wearable sensors have shown that multi-modal deep learning models consistently outperform their uni-modal counterparts.

By Ahmed Mohamady, Robin Burchard, Kristof Van Laerhoven

multimodal benchmarks

arXiv AI

Jun 29

Spectral Text Fusion: A Frequency-Aware Approach to Multimodal Time-Series Forecasting

arXiv:2602. 01588v3 Announce Type: replace-cross Abstract: Multimodal time series forecasting is crucial in real-world applications, where decisions depend on both numerical data and contextual signals.

By Huu Hiep Nguyen, Minh Hoang Nguyen, Dung Nguyen, Hung Le

rag multimodal benchmarks safety

arXiv AI

Jun 29

Chronic Kidney Disease Prognosis Prediction Using Transformer

arXiv:2511. 02340v3 Announce Type: replace Abstract: Chronic Kidney Disease (CKD) affects nearly 10\% of the global population and often progresses to end-stage renal failure.

By Yohan Lee, Dong Gyun Kang, SeHoon Park, Sa-Yoon Park, Kwangsoo Kim

llms nlp fine-tuning efficiency multimodal safety

arXiv AI

Jun 29

What Was That Again? Certified Robustness for Automatic Speech Recognition

arXiv:2606. 27698v1 Announce Type: cross Abstract: Automatic Speech Recognition systems are notoriously both sensitive to adversarial and benign perturbations.

By Andrew C. Cullen, Neil Marchant, Jiani Xie, Paul Montague, Benjamin I. P. Rubinstein

multimodal safety

arXiv AI

Jun 29

NormAct: A Benchmark for Hidden Social Norm Compliance in Embodied Planning

arXiv:2606. 27826v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly deployed as embodied planners in egocentric environments, where task success requires not only achieving instructed goals but also acting in socially appropriate ways.

By Shiyun Zhao, Xinwei Song, Tianyu Guo, Xiaomeng Gao, Mingyuan Liu, Xu Han, Yuanyuan Zhang, Zhenliang Zhang, Xue Feng, Bo Dai

llms agents robotics multimodal benchmarks

arXiv AI

Jun 29

JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications

arXiv:2606. 28070v1 Announce Type: new Abstract: JD.

By Oxygen AIIC, Chan Long, Chao Liu, Chaofan Chen, Chaohui Dong, Chunyuan Guo, Danping Liu, Debin Liu, Deping Xiang, Fulai Xu, Guangyue Liu, Hao Li, Huichun Hu, Jian Yang, Jianan Wang, Jianbo Zhao, Jiaoyang Li, Jiaxing Wang, Jinglong Li, Jinjin Guo, Jun Fang, Jun Liu, Kai Zhou, Li Wang, Lili Gao, Liying Chen, Luning Yang, Mengdi Zhou, Pengzhang Liu, Qi Lv, Qianyun Wang, Qixia Jiang, Ruyue Li, Shimu Liang, Shuxing Wang, Sijie Zhang, Siqi Li, Tianhao Gao, Wang Ke, Weihu Huang, Wencan Lai, Wenjie Zhang, Xiaohui Zhang, Xiaojing Dong, Ya Liu, Yifeng Zhang, Yixiang Wang, Yongtai Zhang, Yongyi Liao, Zhaoru Chen, Zhen Chen, Zhiyong Ma, Zhiyuan Liu, Zhongwei Liu, Ziyan Xing

llms rag efficiency multimodal

arXiv AI

Jun 29

Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook

arXiv:2411. 19537v2 Announce Type: replace-cross Abstract: We survey deepfake generation and detection techniques, covering all deepfake media types: image, video, audio and multimodal content.

By Florinel-Alin Croitoru, Andrei-Iulian Hiji, Vlad Hondru, Nicolae Catalin Ristea, Paul Irofti, Marius Popescu, Cristian Rusu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Mubarak Shah

multimodal benchmarks

arXiv AI

Jun 29

Reflect-R1: Evidence-Driven Reflection for Self-Correction in Long Video Understanding

arXiv:2606. 27922v1 Announce Type: cross Abstract: Current multimodal reflection mechanisms for long video understanding predominantly rely on closed-loop self-reflection within internal parameters.

By Shuimu Chen, Yuteng Chen, Yuanshen Guan, Zebang Cheng, Zeyu Zhang, Shengqian Qin, Bin Xia, Jiaran Li, Wenming Yang, Fei Ma

reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 29

RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance

arXiv:2606. 27766v1 Announce Type: cross Abstract: Offline reinforcement learning enables policy learning from fixed datasets without additional environment interaction, making it appealing for safety-critical applications where online exploration is costly or unsafe.

By Shiqiang Gong

diffusion reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Jun 29

MPFlow: Multi-modal Posterior-Guided Flow Matching for Zero-Shot MRI Reconstruction

arXiv:2603. 03710v3 Announce Type: replace-cross Abstract: Zero-shot MRI reconstruction relies on generative priors, but single-modality unconditional priors produce hallucinations under severe ill-posedness.

By Seunghoi Kim, Chen Jin, Henry F. J. Tregidgo, Matteo Figini, Daniel C. Alexander

diffusion computer-vision multimodal safety

arXiv Machine Learning

Jun 29

EchoSonar-R: A Multi-View Reasoning-Enabled Model for Disease Classification and Report Generation in Echocardiography

arXiv:2606. 28164v1 Announce Type: cross Abstract: Echocardiography is the most widely used non-invasive cardiac imaging modality, providing essential information for cardiovascular diagnosis.

By Darya Taratynova, Ahmed Aly, Numan Saeed, Mohammad Yaqub

llms fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 29

Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings

arXiv:2606. 27543v1 Announce Type: cross Abstract: The variations in vocal effort range (e.

By Zahra Omidi, John H. L. Hansen

multimodal benchmarks

arXiv AI

Jun 29

EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning

arXiv:2603. 09731v3 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly considered as a foundation for embodied agents, yet it remains unclear whether they can reliably reason about the long-term physical consequences of actions from an egocentric viewpoint.

By Chengjun Yu, Xuhan Zhu, Chaoqun Du, Pengfei Yu, Wei Zhai, Yang Cao, Zheng-Jun Zha

llms agents robotics multimodal benchmarks

arXiv AI

Jun 29

Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment

arXiv:2606. 27731v1 Announce Type: cross Abstract: Despite their strong general capabilities, large language models (LLMs) often remain unreliable when outputs must be numerically precise.

By Zhuo Zuo, Li Yue, Wenhao Zheng, Chenpeng Wang, Xianggen Liu

llms nlp multimodal safety

arXiv AI

Jun 29

Home3D 1.0: A High-Fidelity Image-to-3D Asset Generation System for Interior Design

arXiv:2606. 27923v1 Announce Type: cross Abstract: We present Home3D 1.

By Yiyun Fei, Guoqiu Li, Jin Song, Chuqiao Wu, Delong Wu, Hong Wu, Ziru Zeng, Haohui Chen, YinDong Kong, Jing Li, Qi Wu, Feng Zhang

computer-vision multimodal

arXiv AI

Jun 29

Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding

arXiv:2606. 27596v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) exhibit sophisticated reasoning but remain susceptible to object hallucination.

By Liu Yu, Can Chen, Ping Kuang, Zhikun Feng, Fan Zhou, Gillian Dobbie

llms multimodal benchmarks safety

arXiv AI

Jun 29

Parameter Efficient Hybrid Transformer (PEHT) for Network Traffic Prediction via Dynamic Urban Congestion Integration

arXiv:2606. 28274v1 Announce Type: cross Abstract: Accurate network traffic prediction is a critical element for efficient resource allocation in dynamic urban cellular networks.

By Abdolazim Rezaei, Mehdi Sookhak, Mahboobeh Haghparast

llms fine-tuning multimodal benchmarks

Get the daily briefing