Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,303 stories · RSS feed

arXiv Machine Learning

Jul 10

NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL

arXiv:2607. 07855v1 Announce Type: new Abstract: Hierarchical Implicit Q-Learning (HIQL), an offline goal-conditioned RL method, selects subgoals by value-function advantages alone.

By Erdemt Bao, Xing Lei, Jun Chen

reinforcement-learning multimodal safety

arXiv AI

Jul 10

Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models

arXiv:2604. 11530v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have revolutionized multi-modal learning by jointly processing visual and textual information.

By Yvon Apedo, Martyna Poreba, Michal Szczepanski, Samia Bouchafa

llms efficiency multimodal safety

arXiv AI

Jul 10

Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks

arXiv:2607. 07907v1 Announce Type: cross Abstract: With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associations that originate from their training data.

By Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu, Vaidehi Patil

llms multimodal benchmarks

arXiv AI

Jul 10

A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agents

arXiv:2510. 22052v2 Announce Type: replace Abstract: The field of artificial intelligence (AI) has taken a tight hold on broad aspects of society, industry, business, and governance in ways that dictate the prosperity and might of the world's economies.

By Abhijit Chatterjee, Niraj K. Jha, Jonathan D. Cohen, Thomas L. Griffiths, Hongjing Lu, Diana Marculescu, Ashiqur Rasul, Wenrui Xu, Keshab K. Parhi

llms agents multimodal

arXiv AI

Jul 10

Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs

arXiv:2503. 12999v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have demonstrated exceptional performance in various multi-modal tasks.

By Ruichuan An, Kai Zeng, Ming Lu, Sihan Yang, Renrui Zhang, Huitong Ji, Hao Liang, Wentao Zhang

llms fine-tuning multimodal benchmarks data-engineering

arXiv AI

Jul 10

FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigation

arXiv:2607. 08359v1 Announce Type: cross Abstract: Vision-Language Navigation (VLN) enables UAV autonomous navigation in unknown environments by mapping language instructions to real-time visual inputs.

By Xueke Zhu, Qingyan Meng, Liutao Yu, Wei Zhang, Zhengyu Ma, Huihui Zhou, Yonghong Tian

llms agents diffusion multimodal benchmarks

arXiv Machine Learning

Jul 10

A Tool Bottleneck Framework for Clinically-Informed and Interpretable Medical Image Understanding

arXiv:2512. 21414v2 Announce Type: replace-cross Abstract: Recent tool-use frameworks powered by vision-language models (VLMs) improve image understanding by grounding model predictions with specialized tools.

By Christina Liu, Alan Q. Wang, Joy Hsu, Jiajun Wu, Ehsan Adeli

llms multimodal benchmarks

arXiv AI

Jul 10

Aleena: Alignment Agent for Research Software Engineering Collaborations

arXiv:2607. 08043v1 Announce Type: cross Abstract: Research software collaborations span meetings, informal chats, pull requests, and GitHub issues.

By Kshitij Dani, Cordero Core, Landung Setiawan, Carlos Garcia Jurado Suarez, Anshul Tambay, Vani Mandava, Anant Mittal

agents multimodal safety

arXiv AI

Jul 10

WCog-VLA: A Dual-Level World-Cognitive Vision-Language-Action Model for End-to-End Autonomous Driving

arXiv:2607. 08375v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have advanced end-to-end autonomous driving.

By Xuerun Yan, Zhexi Lian, Nuoheng Zhang, Shiyu Fang, Haoran Wang, Chen Lv, Jia Hu, Binyang Song

llms agents diffusion robotics multimodal benchmarks safety

arXiv AI

Jul 10

VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval

arXiv:2607. 08541v1 Announce Type: cross Abstract: Open-vocabulary object detection and segmentation aim to recognize arbitrary objects beyond predefined categories.

By ZhiXin Sun

rag computer-vision nlp multimodal

arXiv AI

Jul 10

SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

arXiv:2607. 08573v1 Announce Type: new Abstract: Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors.

By Adis Alihodzic, Selma Skopljakovic Hubljar

llms multimodal benchmarks

arXiv AI

Jul 10

Best-of-$N$ TTS Evaluation is Confounded by ASR Family Alignment

arXiv:2607. 08256v1 Announce Type: cross Abstract: Best-of-$N$ (BoN) inference improves content consistency in zero-shot text-to-speech by selecting from $N$ candidates with an automatic speech recognition (ASR) verifier.

By Taehyung Yu, Seongjae Kang

multimodal safety

arXiv AI

Jul 10

AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding

arXiv:2607. 08745v1 Announce Type: new Abstract: Recent advances in Vision-Language Models, Large Language Models, and Multimodal Large Language Models have improved autonomous driving tasks such as scene understanding, decision making, trajectory prediction, and visual question answering.

By Siddharth Damodharan, Radhika Gupta, Ali Alshami, Ryan Rabinowitz, Jugal Kalita

llms agents nlp robotics multimodal benchmarks safety

arXiv AI

Jul 10

An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation

arXiv:2602. 13376v2 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly used in document processing pipelines to convert flowchart images into structured code (e.

By Giang Son Nguyen, Zi Pong Lim, Sarthak Ketanbhai Modi, Yon Shin Teo, Wenya Wang

llms computer-vision multimodal

arXiv AI

Jul 10

LEEVLA: Seeing What Matters in Latent Environment Evolution for Vision-Language-Action

arXiv:2607. 08182v1 Announce Type: cross Abstract: Vision-language-action (VLA) models aim to map multimodal inputs to robot actions.

By Qi Lyu, Baicheng Liu, Xudong Wang, Jiahua Dong, Lianqing Liu, Zhi Han

agents diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jul 10

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

arXiv:2607. 08522v1 Announce Type: new Abstract: The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation.

By Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, Leshem Choshen

multimodal benchmarks

arXiv AI

Jul 10

Reinforcing the Generation Order of Multimodal Masked Diffusion Models

arXiv:2607. 08056v1 Announce Type: cross Abstract: Diffusion Language Models (DLMs) have recently achieved substantial progress in natural language generation tasks.

By Yidong Ouyang, Zhe Wang, Sourav Bhabesh, Dmitriy Bespalov

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 10

MatBind: A Shared Embedding Space for Multimodal Materials Characterization

arXiv:2607. 08470v1 Announce Type: new Abstract: Fully characterizing a crystalline material requires integrating heterogeneous data sources -- atomic structures, diffraction patterns, electronic density of states, and natural language -- each of which captures a different facet of the same physical object.

By Le Yang (Institute for Advanced Simulations), Anoop K. Chandran (J\"ulich Supercomputing Centre, Forschungszentrum J\"ulich), Jona \"Ostreicher (Institute of Nanotechnology, Karlsruhe Institute of Technology), Evgenii Sovetkin (J\"ulich Supercomputing Centre, Forschungszentrum J\"ulich), Adrian Mirza (Helmholtz-Zentrum Berlin f\"ur Materialien und Energie, Helmholtz Institute for Polymers in Energy Applications Jena), Sebastien Bompas (Institute for Advanced Simulations), Bashir Kazimi (Institute for Advanced Simulations), Pascal Friederich (Institute of Nanotechnology, Karlsruhe Institute of Technology), Stefan Kesselheim (J\"ulich Supercomputing Centre, Forschungszentrum J\"ulich, 1. Physikalisches Institut, University of Cologne), Kevin Maik Jablonka (Helmholtz Institute for Polymers in Energy Applications Jena, Center for Energy and Environmental Chemistry Jena, Friedrich Schiller University Jena), Stefan Sandfeld (Institute for Advanced Simulations, Faculty 5 - Georesources and Materials Engineering, RWTH Aachen University)

rag multimodal safety

arXiv Machine Learning

Jul 10

Expressivity and Statistical Trade-offs in Diffusion Policy Learning

arXiv:2607. 07967v1 Announce Type: cross Abstract: Diffusion-based policies have recently emerged as powerful policy parameterizations for reinforcement learning, representing state-conditioned action distributions as terminal laws of diffusion processes with parameterized drifts.

By Viet Vu, Renyuan Xu, Jiacheng Zhang, Yufei Zhang

diffusion reinforcement-learning multimodal

arXiv AI

Jul 10

MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulation

arXiv:2510. 07328v2 Announce Type: replace-cross Abstract: Medical decision systems increasingly rely on data from multiple sources to ensure reliable and unbiased diagnosis.

By Md Zubair, Hao Zheng, Grayson W. Armstrong, Lucy Q. Shen, Gabriela Wilson, Yu Tian, Xingquan Zhu

multimodal safety

Get the daily briefing