Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,277 stories · RSS feed

arXiv Machine Learning

Jun 16

LLM-Based Synthetic Ground Truth Generation for Audio-Based Emotion Classification via In-Context Learning

arXiv:2606. 14784v1 Announce Type: cross Abstract: Understanding human states and interaction dynamics is a core goal of human-computer interaction (HCI).

By Qing Huang, Pooja Pol, Jianing Zhang

llms agents fine-tuning multimodal

arXiv Machine Learning

Jun 16

The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints

arXiv:2606. 16028v1 Announce Type: new Abstract: Modern deep learning architectures are increasingly multi-task and multi-modal, using a pretrained foundation model combined with task-specific, fine-tuned models.

By Thomas Dittrich, Oliver Potocki, Philipp Grohs

fine-tuning multimodal

arXiv Machine Learning

Jun 16

REFLEX: Reflective Evolution from LLM Experience

arXiv:2606. 16496v1 Announce Type: cross Abstract: Large multimodal language models (LLMs) have emerged as powerful tools for guiding evolutionary search toward interpretable programmatic policies.

By Pan Wang

llms multimodal benchmarks

arXiv AI

Jun 16

Teacher-Student Structure for Domain Adaptation in Ensemble Audio-Visual Video Deepfake Detection

arXiv:2606. 15117v1 Announce Type: cross Abstract: The rapid advancement of generative AI models is leading to more realistic deepfake media, encompassing the manipulation of audio, video, or both.

By Elham Abolhasani, Maryam Ramezani, Hamid R. Rabiee

robotics fine-tuning multimodal safety

arXiv AI

Jun 16

Multi-Modal Attention for Automated Disaster Damage Assessment Using Remote Sensing Imagery and Deep Learning

arXiv:2606. 14963v1 Announce Type: cross Abstract: Timely and accurate disaster damage assessment is crucial for effective emergency response, resource allocation, and recovery.

By Tewodros Syum Gebre, Jagrati Talreja, Leila Hashemi-Beni

multimodal

arXiv AI

Jun 16

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

arXiv:2606. 15749v1 Announce Type: cross Abstract: Traffic scene understanding requires models to reason beyond object recognition, including lane topology, multi-view geometry, temporal evolution, and signal-phase semantics.

By Maonan Wang, Zhengyan Huang, Kemou Jiang, Yuhang Fu, Jiayue Zhu, Yuxin Cai, Xingchen Zou, Qiaosheng Zhang, Yi Yu, Ding Wang, Xi Chen, Ben M. Chen, Yuxuan Liang, Zhiyong Cui, Man On Pun, Yirong Chen

fine-tuning multimodal benchmarks

arXiv AI

Jun 16

MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

arXiv:2606. 14747v1 Announce Type: cross Abstract: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs).

By Haitian Wang, Ruoxi Sun, Quantong Qiu, Juntao Li, Junhui Li, Hua Chen, Jinxiong Chang, Min Zhang

rag multimodal benchmarks

arXiv AI

Jun 16

FusionRS: A Large-Scale RGB-Infrared Remote Sensing Dataset for Dual-Modal Vision-Language Foundation Models

arXiv:2606. 17020v1 Announce Type: cross Abstract: Remote sensing vision-language models have advanced Earth observation understanding, but most existing work remains centered on RGB imagery, leaving the complementary information in infrared data underexplored.

By Jiaju Han, Ben Zhang, Xuemeng Sun, Qike Zhang, Yuxian Dong, Chengyin Hu, Fengyu Zhang, Yiwei Wei, Jiujiang Guo

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 16

OmniMouse: Scaling properties of multi-modal, multi-task Brain Models on 150B Neural Tokens

arXiv:2604. 18827v2 Announce Type: replace-cross Abstract: Scaling data and artificial neural networks has transformed AI, driving breakthroughs in language and vision.

By Konstantin F. Willeke, Polina Turishcheva, Alex Gilbert, Goirik Chakrabarty, Hasan A. Bedel, Paul G. Fahey, Yongrong Qiu, Marissa A. Weis, Michaela Vystr\v{c}ilov\'a, Taliah Muhammad, Lydia Ntanavara, Rachel E. Froebe, Kayla Ponder, Zheng Huan Tan, Emin Orhan, Erick Cobos, Sophia Sanborn, Katrin Franke, Fabian H. Sinz, Alexander S. Ecker, Andreas S. Tolias

llms computer-vision multimodal benchmarks

arXiv AI

Jun 16

NVMOS: Non-Verbal Vocalization Quality Assessment in Speech

arXiv:2606. 15888v1 Announce Type: cross Abstract: Non-verbal vocalizations (NVs), such as laughter, sighs, and coughs, are important acoustic cues for emotion and intent.

By Jialong Mai, Jinxin Ji, Xiaofen Xing, Wencui Liu, Xiangmin Xu

llms multimodal

arXiv AI

Jun 16

Virtual Sensing to Enable Real-Time Monitoring of Inaccessible Locations & Unmeasurable Parameters

arXiv:2412. 00107v2 Announce Type: replace-cross Abstract: Real-time monitoring of safety-critical interior states remains an open problem in energy systems where physical instrumentation is infeasible.

By Kazuma Kobayashi, Farid Ahmed, Jaewan Park, Subhankar Sarkar, Souvik Chakraborty, Syed Bahauddin Alam

multimodal safety

arXiv Machine Learning

Jun 16

A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT

arXiv:2606. 16991v1 Announce Type: cross Abstract: Multiphasic contrast-enhanced CT (CECT) is widely used for abdominal lesion characterization, yet it carries inherent risks of contrast-induced nephropathy, escalates acquisition burden, and heavily contributes to radiologist workload.

By Mariam Elbakry, Aliaa Sayed Sheha, Salma Hassan Tantawy, Aya Yassin, Concetto Spampinato, Karim Lekadir, Xiaomeng Li, Marawan Elbatel

multimodal benchmarks

arXiv AI

Jun 16

Wasserstein Equilibrium Decoding for Reliable Medical Visual Question Answering

arXiv:2605. 18313v2 Announce Type: replace-cross Abstract: Small vision-language models (2-8B) are well-suited for clinical deployment due to privacy constraints, limited connectivity, and low-latency requirements favouring on-device or on-premise inference.

By Luca Hagen, Johanna P. M\"uller, Weitong Zhang, Mengyun Qiao, Bernhard Kainz

llms nlp fine-tuning multimodal safety

arXiv AI

Jun 16

Attention, not scale, drives human-AI alignment in multimodal language prediction

arXiv:2308. 06035v4 Announce Type: replace Abstract: Humans routinely draw on visual context to predict upcoming words.

By Viktor Kewenig, Andrew Lampinen, Samuel A. Nastase, Christopher Edwards, Quitterie Lacome D'Elascombe, Akilles Rechardt, Jeremy I Skipper, Gabriella Vigliocco

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 16

DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning

arXiv:2606. 15160v1 Announce Type: cross Abstract: Reasoning capabilities of multimodal large language models (MLLMs) have improved considerably in recent years.

By David Huang, Lianlei Shan

llms diffusion reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Jun 16

Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screening

arXiv:2606. 14788v1 Announce Type: cross Abstract: Voice-based screening offers a scalable and non-invasive way to assess neurodegenerative diseases such as Alzheimer's disease (AD) and Parkinson's disease (PD), but their staging remains challenging due to the difficulty of integrating heterogeneous data.

By Qingfeng Zhang, Yuanxiong Guo, Yanmin Gong

llms rag multimodal

arXiv AI

Jun 16

CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions

arXiv:2512. 01095v2 Announce Type: replace-cross Abstract: We present CycliST, a novel benchmark dataset designed to evaluate Video Language Models (VLM) on their ability for textual reasoning over cyclical state transitions.

By Simon Kohaut, Daniel Ochs, Shun Zhang, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami

llms multimodal benchmarks

arXiv AI

Jun 16

Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling

arXiv:2606. 15038v1 Announce Type: new Abstract: Accurate time-to-event (TTE) prediction from multimodal clinical data remains challenging due to modality imbalance and distribution shift.

By Zhemin Zhang, Weijie Chen, David Le, Amara Tariq, Alex Wallace, Matthew Stib, Juan Maria Farina, Chadi Ayoub, Reza Arsanjani, Imon Banerjee

diffusion multimodal safety

arXiv AI

Jun 16

Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling

arXiv:2606. 15623v1 Announce Type: cross Abstract: Pairwise comparison is the gold standard for subjective ranking tasks; however, exhaustive annotation requires a massive number of human comparisons ($O(n^2)$).

By Yujin Park, Haejun Chung, Ikbeom Jang

llms multimodal benchmarks safety

arXiv AI

Jun 16

Akasha 2: Hamiltonian State Space Duality and Visual-Language Joint Embedding Predictive Architectur

arXiv:2601. 06212v2 Announce Type: replace-cross Abstract: We present Akasha 2, a state-of-the-art multimodal architecture that integrates Hamiltonian State Space Duality (H-SSD) with Visual-Language Joint Embedding Predictive Architecture (VL-JEPA).

By Yani Meziani

llms rag diffusion multimodal benchmarks

Get the daily briefing