Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,964 stories · RSS feed

Hugging Face Trending Papers

Jul 10

AnythingReality: Robust Online Gaussian Splatting SLAM for Open-Vocabulary VR Scene Exploration

We present a novel integrated architecture for robust online 3D Gaussian splatting, real-time VR exploration, and speech-driven Vision-Language-Model interaction. Unlike methods assuming clean depth or external poses, our system combines ORB-SLAM3-based pose estimation with online Gaussian reconstruction for noisy real-world data.

computer-vision multimodal benchmarks

arXiv AI

Jul 10

Omni-Sleep: A Sleep Foundation Model via Hierarchical Contrastive Learning of CNS--ANS Dynamic

arXiv:2607. 07720v1 Announce Type: cross Abstract: Sleep physiology arises from the coordinated dynamics of the central nervous system (CNS) and autonomic nervous system (ANS), as reflected by multimodal polysomnography signals including EEG, EOG, EMG, ECG, and respiration.

By Zhoujie Hou, Song Wang, Kexin Lou, Mo Wang, Chen Wei, Quanying Liu

multimodal benchmarks

arXiv Machine Learning

Jul 10

A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding

arXiv:2512. 21414v2 Announce Type: replace-cross Abstract: Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools.

By Christina Liu, Alan Q. Wang, Joy Hsu, Jiajun Wu, Ehsan Adeli

llms multimodal benchmarks

arXiv AI

Jul 10

Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

arXiv:2607. 08317v1 Announce Type: new Abstract: Modern AI models achieve strong performance on many established benchmarks, yet they still fail on tasks that humans find almost trivial, such as manipulating a string or drawing a dog with five legs.

By Matteo Santelmo, Xiuying Wei, Israa Fakih, Felix Bauer, Juan Garcia Giraldo, Chengkun Li, Etienne Bamas, Emmanuel Abb\'e

multimodal benchmarks

arXiv AI

Jul 10

Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs

arXiv:2607. 08193v1 Announce Type: cross Abstract: Open-ended curricula in Reinforcement Learning (RL) aim to train generally-capable agents by identifying tasks that facilitate learning increasingly complex skills.

By Lorenzo Pant\`e, Andrea Fanti, Roberto Capobianco

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 10

LDFE: Laplacian Decoupled Feature Enhancement Block for Dual-Stream CNN-based RGB-IR Object Detection

arXiv:2607. 08076v1 Announce Type: cross Abstract: The complementary information between RGB and IR images can significantly enhance object detection performance under extreme conditions.

By Wenhao Dong, Xiaoyan Luo, Linlin Yang, Haodong Zhu, Xiaorong Shi, Guodong Guo, Baochang Zhang

computer-vision multimodal benchmarks

arXiv Machine Learning

Jul 10

Cross-Modal Generative Framework for Signal Translation from Fetal-Maternal Electrocardiograms to Fetal Doppler Waveforms

arXiv:2607. 08073v1 Announce Type: new Abstract: Fetal electrocardiogram (fECG) and Doppler ultrasound provide complementary views of fetal cardiovascular function: fECG captures electrical activity while Doppler reflects mechanical hemodynamics shaped by factors such as placental resistance and vascular compliance.

By Tongli Su, Alireza Rafiei, Marly van Assen, Reza Sameni, Gari D. Clifford, Faezeh Marzbanrad, Nasim Katebi

multimodal

arXiv AI

Jul 10

OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice

arXiv:2607. 08423v1 Announce Type: new Abstract: The rapid integration of Large Vision-Language Models (VLMs) into critical infrastructure promises to revolutionize personalized healthcare and dietary management.

By Qian Jiang, Zhecheng Shi, Jingpu Yang, Zirui Song, Miao Fang

llms agents multimodal benchmarks safety

arXiv AI

Jul 10

Aleena: Alignment Agent for Research Software Engineering Collaborations

arXiv:2607. 08043v1 Announce Type: cross Abstract: Research software collaborations span meetings, informal chats, pull requests, and GitHub issues.

By Kshitij Dani, Cordero Core, Landung Setiawan, Carlos Garcia Jurado Suarez, Anshul Tambay, Vani Mandava, Anant Mittal

agents multimodal safety

arXiv AI

Jul 10

An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation

arXiv:2602. 13376v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly used in document processing pipelines to convert flowchart images into structured code (e.

By Giang Son Nguyen, Zi Pong Lim, Sarthak Ketanbhai Modi, Yon Shin Teo, Wenya Wang

llms computer-vision multimodal

arXiv AI

Jul 10

Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models

arXiv:2604. 11530v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have revolutionized multi-modal learning by jointly processing visual and textual information.

By Yvon Apedo, Martyna Poreba, Michal Szczepanski, Samia Bouchafa

llms efficiency multimodal safety

arXiv AI

Jul 10

APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts

arXiv:2607. 08024v1 Announce Type: cross Abstract: Long-horizon robot planning requires jointly reasoning over semantic task structure and geometric feasibility.

By Emily Jin, Joy Hsu, Yiqing Xu, Weiyu Liu, Nick Haber, Jiajun Wu

robotics multimodal

arXiv AI

Jul 10

SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

arXiv:2607. 08573v1 Announce Type: new Abstract: Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors.

By Adis Alihodzic, Selma Skopljakovic Hubljar

llms multimodal benchmarks

arXiv Machine Learning

Jul 10

The Importance of Encoder Choice:A Tabular-Image Study

arXiv:2607. 07756v1 Announce Type: new Abstract: Multimodal learning usually requires a dedicated encoder per modality.

By Ilia Koloiarov, Diego Coello de Portugal Mecke, Vijaya Krishna Yalavarthi, Tom Hanika, Lars Schmidt-Thieme

llms multimodal benchmarks

arXiv AI

Jul 10

Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction

arXiv:2607. 08233v1 Announce Type: new Abstract: A central challenge in building intelligent systems is enabling agents to jointly perceive complex inputs, form hypotheses about hidden patterns, and design informative experiments to test them.

By Sophia Koehler, Antonia W\"ust, Inga Ibs, Wasu Top Piriyakulkij, Wolfgang Stammer, Constantin Rothkopf, Kevin Ellis, Kristian Kersting

agents multimodal

arXiv AI

Jul 10

PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

arXiv:2607. 08111v1 Announce Type: cross Abstract: Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable.

By Wanyi Ning, Wei Zhou, Yingpeng Li, Yinshang Guo, Haitao Qian, Yiming Cheng

fine-tuning multimodal benchmarks

arXiv AI

Jul 10

Pose-to-Biomechanics: Bridging 3D Human Pose Estimation and Biomechanical Attribute Prediction

arXiv:2607. 08725v1 Announce Type: cross Abstract: Recent progress in 3D human pose estimation has made markerless recovery of skeletal motion increasingly accurate and scalable.

By Ayda Eghbalian, Kevin Desai

llms computer-vision multimodal benchmarks

arXiv AI

Jul 10

Infinity-Parser2 Technical Report

arXiv:2607. 07836v1 Announce Type: new Abstract: We present Infinity-Parser2, a large multimodal model that couples a controllable data-synthesis pipeline with multi-task reinforcement learning for end-to-end document parsing, addressing the persistent scarcity of faithfully annotated parsing corpora.

By Zuming Huang, Jun Huang, Kexuan Ren, Baode Wang, Weizhen Li, Jianming Feng, Yu Wang, Yichen Yao, Shijun Lin, Yige Tang, Cheng Peng, Weidi Xu, Wei Chu, Yinghui Xu, Yuan Qi

computer-vision reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Jul 10

VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval

arXiv:2607. 08541v1 Announce Type: cross Abstract: Open-vocabulary object detection and segmentation aim to recognize arbitrary objects beyond predefined categories.

By ZhiXin Sun

rag computer-vision nlp multimodal

arXiv Machine Learning

Jul 10

Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix

arXiv:2607. 08312v1 Announce Type: new Abstract: How should language interface with a world model's discrete symbol system?

By Jiayi Fang

llms robotics fine-tuning multimodal

Get the daily briefing