Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,022 stories · RSS feed

arXiv AI

Jul 24

OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model

arXiv:2602. 12304v5 Announce Type: replace-cross Abstract: Existing mainstream video customization methods focus on generating identity-consistent videos based on given reference images and textual prompts.

By Maomao Li, Zhen Li, Kaipeng Zhang, Guosheng Yin, Zhifeng Li, Dong Xu

diffusion fine-tuning multimodal

arXiv AI

Jul 24

MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs

arXiv:2508. 05502v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) perform strongly in high-resource languages, yet often produce fluent but culturally "thin" descriptions in low-resource settings.

By Yufei Gao, Jiaying Fei, Nuo Chen, Ruirui Chen, Guohang Yan, Yunshi Lan, Botian Shi

llms fine-tuning multimodal safety

arXiv AI

Jul 24

Faster IndexTTS-2: Accelerating and Streaming Autoregressive Zero-Shot Text-to-Speech Synthesis on GPUs

arXiv:2607. 21042v1 Announce Type: new Abstract: Autoregressive text-to-speech models achieve strong naturalness but suffer from slow inference due to sequential token generation, limiting their deployment in production applications that require low latency.

By Muyang Du, Shuang Yu, Junjie Lai

llms diffusion efficiency multimodal benchmarks

arXiv Machine Learning

Jul 24

Adaptive Confidence-weighted Expansion for Trustworthy Multi-Omics Multimodal Fusion

arXiv:2607. 20742v1 Announce Type: new Abstract: Multimodal learning is a robust approach to improve predictive performance in applications such as medical prognosis.

By Mohammad Raahemi, Ali Sekhavati, Alireza Maleki, Hamid Nasiri

multimodal benchmarks safety data-engineering

arXiv Machine Learning

Jul 24

Environment-Aware Channel Inference via Cross-Modal Flow: From Multimodal Sensing to Wireless Channels

arXiv:2512. 04966v2 Announce Type: replace-cross Abstract: Accurate channel state information (CSI) underpins reliable and efficient wireless communication.

By Guangming Liang, Mingjie Yang, Dongzhu Liu, Paul Henderson, Lajos Hanzo

diffusion multimodal benchmarks safety

arXiv AI

Jul 24

TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

arXiv:2602. 19313v2 Announce Type: replace-cross Abstract: General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior.

By Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, Ranjay Krishna

llms robotics multimodal benchmarks

arXiv AI

Jul 24

VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memory

arXiv:2603. 04910v2 Announce Type: replace-cross Abstract: Imitation learning from human demonstrations has achieved significant success in robotic control, yet most visuomotor policies still condition on single-step observations or short-context histories, making them struggle with non-Markovian tasks that require long-term memory.

By Yuheng Lei, Zhixuan Liang, Hongyuan Zhang, Ping Luo

llms rag diffusion robotics multimodal benchmarks

arXiv AI

Jul 24

Vision-Language-Policy Model for Dynamic Robot Task Planning

arXiv:2512. 19178v2 Announce Type: replace-cross Abstract: Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics.

By Jin Wang, Kim Tien Ly, Jacques Cloete, Jin Jin, Nikos Tsagarakis, Ioannis Havoutis

llms agents robotics fine-tuning multimodal

arXiv Machine Learning

Jul 24

Best-of-Evidence: Best-of-N Selection under Partial Verification

arXiv:2607. 20950v1 Announce Type: new Abstract: BoN improves model outputs by sampling several candidates and selecting one with a proxy score, but it assumes that complete candidates can be evaluated reliably.

By Cenwei Zhang, Teng Fang, Yuxia Wang, Derek Li, Bryan Dai, Lei You

multimodal

arXiv AI

Jul 24

Monkey King Bang: A Unified Scientific Multimodal Foundation Model

arXiv:2607. 20557v1 Announce Type: cross Abstract: Scientific discovery is increasingly shifting from isolated disciplines to multi-domain reasoning, and AI for science faces a similar transition.

By Hesen Chen, Xinyu Su, Xiaomeng Yang, Yuetan Lin, Zixiong Yang, Junyi An, Fenglei Cao, Yifeng Jiao, Yunqi Zhang, Yuan Cheng, Zhiyu Tan, Hao Li, Libo Wu, Yuan Qi

llms computer-vision multimodal benchmarks

arXiv Machine Learning

Jul 24

Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs

arXiv:2607. 20517v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) over heterogeneous PDF collections remains challenging due to multimodal content, domain-specific terminology, and the need for multi-hop reasoning across dispersed evidence.

By Takato Yasuno

llms rag computer-vision multimodal benchmarks safety

arXiv AI

Jul 24

Multimodal Learning for Arcing Detection in Pantograph-Catenary Systems

arXiv:2602. 08792v2 Announce Type: replace-cross Abstract: The pantograph-catenary interface is essential for ensuring uninterrupted and reliable power delivery in electrified rail systems.

By Hao Dong, Eleni Chatzi, Olga Fink

multimodal benchmarks

arXiv AI

Jul 24

Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain

arXiv:2607. 20510v1 Announce Type: new Abstract: We introduce Telco-GAIA, a bilingual, multi-modal benchmark for evaluating tool-using agents on the data of a real-world telecommunications operator.

By Dmitrii Khizbullin, Zaid Alyafeai, Abdelrahman Eldesokey, Nourah AlSultan, Raghad Alshalan, David R. Pugh, Bernard Ghanem

llms agents multimodal benchmarks data-engineering

arXiv AI

Jul 24

EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization

arXiv:2607. 21013v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved impressive performance in multimodal emotion recognition (MER) tasks and lifted MER to a new level that is complex emotion understanding with advanced video understanding abilities and natural language description.

By Lihuang Fang, Yuchen Zou, kebin Jin, Jinghui Qin

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 24

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

arXiv:2607. 21300v1 Announce Type: cross Abstract: Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations.

By Lorenzo Orsingher, Thomas De Min, Massimiliano Mancini, Davide Talon, Elisa Ricci

llms nlp fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving

arXiv:2607. 20988v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models augmented with world modeling represent a promising paradigm for end-to-end autonomous driving.

By Quanfu Yu, Xian Wu, Hao Xu, Liulong Ma

agents robotics fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jul 24

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

arXiv:2607. 21496v1 Announce Type: cross Abstract: Cognitive impairment (CI) is a growing public health concern.

By Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

llms rag multimodal benchmarks safety

arXiv AI

Jul 24

3D-Aware VLMs with Implicit and Explicit Geometries

arXiv:2607. 21595v1 Announce Type: cross Abstract: Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning.

By Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Ran Xu, Shijian Lu, Gongjie Zhang

llms multimodal

arXiv AI

Jul 24

M$^3$-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data

arXiv:2607. 21343v1 Announce Type: cross Abstract: Integrating heterogeneous biomedical data, including clinical metadata, histopathology images, and molecular profiles, is crucial for comprehensive disease understanding.

By Francesca Pia Panaccione, Carlo Sgaravatti, Marco Venere

rag multimodal safety

arXiv AI

Jul 24

GS-Agent: Creating 4D Physical Worlds With Generative Simulation

arXiv:2607. 21522v1 Announce Type: cross Abstract: Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging.

By Hongxin Zhang, Chunru Lin, Junyan Li, Zhou Xian, Tsun-Hsuan Wang, Chuang Gan

agents robotics fine-tuning multimodal

Get the daily briefing