Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,022 stories · RSS feed

arXiv AI

Jul 24

Faster IndexTTS-2: Accelerating and Streaming Autoregressive Zero-Shot Text-to-Speech Synthesis on GPUs

arXiv:2607. 21042v1 Announce Type: new Abstract: Autoregressive text-to-speech models achieve strong naturalness but suffer from slow inference due to sequential token generation, limiting their deployment in production applications that require low latency.

By Muyang Du, Shuang Yu, Junjie Lai

llms diffusion efficiency multimodal benchmarks

arXiv AI

Jul 24

Robostral Navigate

arXiv:2607. 20785v1 Announce Type: cross Abstract: Deploying navigation systems at scale requires a recipe that minimizes sensor assumptions, generalizes across robot embodiments, and trains efficiently.

By Arjun Majumdar, Avinash Sooriyarachchi, Benjamin Tibi, Chris Bamford, Elliot Chane-Sane, Guillaume Lample, Khyathi Raghavi Chandu, Ludovic Ho Fuh, Mathieu Poiree, Olivier Duchenne, Rosalie Millner, Srijan Mishra, Theo Cachet, Thomas Chabal

llms reinforcement-learning robotics multimodal benchmarks

arXiv AI

Jul 24

OpenForgeRL: Train Harness-native Agents in Any Environment

arXiv:2607. 21557v1 Announce Type: new Abstract: Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems.

By Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jul 24

OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model

arXiv:2602. 12304v5 Announce Type: replace-cross Abstract: Existing mainstream video customization methods focus on generating identity-consistent videos based on given reference images and textual prompts.

By Maomao Li, Zhen Li, Kaipeng Zhang, Guosheng Yin, Zhifeng Li, Dong Xu

diffusion fine-tuning multimodal

arXiv AI

Jul 24

MIRROR: Learning from the Other View for Multi-Modal Reasoning

arXiv:2607. 21552v1 Announce Type: new Abstract: Unlike large language models (LLMs) that exhibit strong reasoning capabilities, vision-language models (VLMs) struggle with visual reasoning, even on geometry problems that admit equivalent text, diagram, and combined diagram+text views.

By Wen Ye, Yuxiao Qu, Aviral Kumar, Xuezhe Ma

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jul 24

Multimodal Pretraining for Generalizable EEG Representation Learning

arXiv:2607. 21384v1 Announce Type: new Abstract: Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks.

By Targol Bakhtiarvand, Jugal Kalita, Adham Atyabi

llms rag computer-vision fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

arXiv:2607. 21300v1 Announce Type: cross Abstract: Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations.

By Lorenzo Orsingher, Thomas De Min, Massimiliano Mancini, Davide Talon, Elisa Ricci

llms nlp fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

OPOD: On-Policy Omni Distillation

arXiv:2607. 20918v1 Announce Type: new Abstract: Omni-modal models can handle text, images, and audio in one system, but improving all of these abilities together remains difficult.

By Tong Zhao, Yuyang Hu, Reed Li, Yu Lu, Haibo Shi, Yutao Zhu, Zhicheng Dou

efficiency multimodal benchmarks

arXiv Machine Learning

Jul 24

Adaptive Confidence-weighted Expansion for Trustworthy Multi-Omics Multimodal Fusion

arXiv:2607. 20742v1 Announce Type: new Abstract: Multimodal learning is a robust approach to improve predictive performance in applications such as medical prognosis.

By Mohammad Raahemi, Ali Sekhavati, Alireza Maleki, Hamid Nasiri

multimodal benchmarks safety data-engineering

arXiv AI

Jul 24

Multimodal Learning for Arcing Detection in Pantograph-Catenary Systems

arXiv:2602. 08792v2 Announce Type: replace-cross Abstract: The pantograph-catenary interface is essential for ensuring uninterrupted and reliable power delivery in electrified rail systems.

By Hao Dong, Eleni Chatzi, Olga Fink

multimodal benchmarks

arXiv AI

Jul 24

Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts

arXiv:2607. 20462v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into clinical workflows, stressing the need for reliable traceability of model-generated output with watermarking.

By Melanie Rieff, Robin Staab, Thibaud Gloaguen, Stefan Hegselmann, Martin Vechev

llms multimodal benchmarks safety

arXiv AI

Jul 24

Sparse Concept Channels in Frozen 3D CT Vision Encoders

arXiv:2607. 20993v1 Announce Type: cross Abstract: Large vision-language models are becoming increasingly dominant in 3D medical image interpretation, but we rarely know which internal units encode clinical findings or where that information lives in the representation.

By Farhad Nooralahzadeh, Lea Bogensperger, Christian Bluethgen, Michael Krauthammer

llms rag multimodal

arXiv AI

Jul 24

Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain

arXiv:2607. 20510v1 Announce Type: new Abstract: We introduce Telco-GAIA, a bilingual, multi-modal benchmark for evaluating tool-using agents on the data of a real-world telecommunications operator.

By Dmitrii Khizbullin, Zaid Alyafeai, Abdelrahman Eldesokey, Nourah AlSultan, Raghad Alshalan, David R. Pugh, Bernard Ghanem

llms agents multimodal benchmarks data-engineering

arXiv AI

Jul 24

MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs

arXiv:2508. 05502v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) perform strongly in high-resource languages, yet often produce fluent but culturally "thin" descriptions in low-resource settings.

By Yufei Gao, Jiaying Fei, Nuo Chen, Ruirui Chen, Guohang Yan, Yunshi Lan, Botian Shi

llms fine-tuning multimodal safety

arXiv AI

Jul 24

WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidance

arXiv:2511. 12997v2 Announce Type: replace Abstract: Multimodal LLM-powered agents have recently demonstrated impressive capabilities in web navigation, enabling agents to complete complex browsing tasks across diverse domains.

By Genglin Liu, Shijie Geng, Sha Li, Hejie Cui, Sarah Zhang, Xin Liu, Tianyi Liu

llms agents multimodal benchmarks

arXiv AI

Jul 24

M$^3$-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data

arXiv:2607. 21343v1 Announce Type: cross Abstract: Integrating heterogeneous biomedical data, including clinical metadata, histopathology images, and molecular profiles, is crucial for comprehensive disease understanding.

By Francesca Pia Panaccione, Carlo Sgaravatti, Marco Venere

rag multimodal safety

arXiv Machine Learning

Jul 24

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

arXiv:2607. 21496v1 Announce Type: cross Abstract: Cognitive impairment (CI) is a growing public health concern.

By Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

llms rag multimodal benchmarks safety

arXiv Machine Learning

Jul 24

Multimodality Stacking with Blockwise missing values and application to the PIONeeR biomarkers study for prediction of resistance to immunotherapy

arXiv:2605. 25050v2 Announce Type: replace-cross Abstract: Integrating multimodal datasets in clinical oncology is frequently hindered by high dimensionality and blockwise missingness, where entire data sources are unavailable for specific patient subsets.

By Mohamed Boussena, Florence Monville, Jacques Fieschi-Meric, Frederic Vely, Pierre Milpied, Julien Mazieres, Maurice Perol, Eric Vivier, Laurent Greillier, Fabrice Barlesi, Sebastien Benzekry

multimodal

arXiv AI

Jul 24

Monkey King Bang: A Unified Scientific Multimodal Foundation Model

arXiv:2607. 20557v1 Announce Type: cross Abstract: Scientific discovery is increasingly shifting from isolated disciplines to multi-domain reasoning, and AI for science faces a similar transition.

By Hesen Chen, Xinyu Su, Xiaomeng Yang, Yuetan Lin, Zixiong Yang, Junyi An, Fenglei Cao, Yifeng Jiao, Yunqi Zhang, Yuan Cheng, Zhiyu Tan, Hao Li, Libo Wu, Yuan Qi

llms computer-vision multimodal benchmarks

Hugging Face Trending Papers

Jul 23

MIRROR: Learning from the Other View for Multi-Modal Reasoning

Unlike large language models (LLMs) that exhibit strong reasoning capabilities, vision-language models (VLMs) struggle with visual reasoning, even on geometry problems that admit equivalent text, diagram, and combined diagram+text views. We show that these views often elicit different behaviors: a model may solve a problem from text but fail on the corresponding diagram, or succeed visually while failing textually.

llms reinforcement-learning multimodal benchmarks

Get the daily briefing