Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,303 stories · RSS feed

arXiv Machine Learning

Jul 13

Variable-Length Generative Protein Design via Generalized Poisson Flow

arXiv:2607. 09039v1 Announce Type: new Abstract: The ability to generate variable-length proteins is crucial in protein design, where the optimal length is often unknown and tightly coupled to designability.

By Chaoran Cheng, Zhanghan Ni, Yanru Qu, Yuxin Chen, Ruihan Guo, Jiajun Fan, Ge Liu

diffusion multimodal

arXiv Machine Learning

Jul 13

Robustifying Vision-Language Models via Test-Time Prompt Adaptation

arXiv:2607. 09450v1 Announce Type: cross Abstract: Pre-trained Vision-Language Models (VLMs) such as CLIP achieve strong zero-shot generalization, but their performance degrades sharply under adversarial perturbations.

By Xingyu Zhu, Huanshen Wu, Shuo Wang, Beier Zhu, Jiannan Ge, Jiaheng Zhang, Long Chen

llms multimodal safety

arXiv AI

Jul 13

SAGEAgent: A Self-Evolving Agent for Cost-Aware Modality Acquisition in Multimodal Survival Prediction

arXiv:2607. 09521v1 Announce Type: new Abstract: Does every cancer patient truly need a complete diagnostic workup for accurate survival prediction?

By Chongyu Qu, Can Cui, Zhengyi Lu, Junchao Zhu, Tianyuan Yao, Junlin Guo, Juming Xiong, Yanfan Zhu, Yuechen Yang, Bennett A. Landman, Yuankai Huo

llms agents multimodal

arXiv AI

Jul 13

ConceptSMILE: Auditing the Trustworthiness of Concept-Based Explainable AI

arXiv:2607. 09649v1 Announce Type: new Abstract: Concept-based explainable artificial intelligence (AI) can make model reasoning more human-understandable, but concept-level outputs are not automatically trustworthy.

By Mohadeseh Mollapour, Koorosh Aslansefat, Zeinab Dehghani, Bhupesh Kumar Mishra, Tejal Shah, Zhibao Mian

multimodal

arXiv AI

Jul 13

Task-Specific Multimodal Question Answering Agents via Confidence Calibration and Incremental Reasoning for QANTA 2026

arXiv:2607. 09623v1 Announce Type: cross Abstract: We present our submission to the QANTA 2026 shared challenge at the ICML 2026 Workshop on Efficient Multimodal Question Answering (EMM-QA).

By Nirjhar Das, Md. Al-Mamun Provath

llms agents nlp multimodal benchmarks

arXiv AI

Jul 13

SVF-CR: Synchronized Visual-Facial Cross-Refinement for Multimodal Ambivalence and Hesitancy Recognition

arXiv:2607. 09417v1 Announce Type: cross Abstract: Ambivalence and hesitancy are subtle behavioral states that are expressed through a combination of verbal content, facial behavior, visual context, and acoustic cues.

By Hyein Park, Namho Kim, Junhwa Kim

multimodal

arXiv AI

Jul 13

M4V: Multimodal Mamba for Efficient Text-to-Video Generation

arXiv:2506. 10915v2 Announce Type: replace-cross Abstract: Text-to-video generation has significantly enriched content creation and holds the potential to evolve into powerful world simulators.

By Jiancheng Huang, Gengwei Zhang, Zequn Jie, Siyu Jiao, Yinlong Qian, Ling Chen, Yunchao Wei, Lin Ma

llms diffusion multimodal benchmarks

arXiv AI

Jul 13

Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

arXiv:2605. 28215v2 Announce Type: replace Abstract: In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labelled examples.

By Carmen Quiles-Ram\'irez, Leticia L. Rodr\'iguez, Nicol\'as Martorell, Natalia D\'iaz-Rodr\'iguez

llms multimodal benchmarks

arXiv AI

Jul 13

Decoupling Language Guidance from Backbones for Text-Guided Medical Segmentation

arXiv:2607. 09481v1 Announce Type: cross Abstract: Text-guided medical image segmentation leverages clinical semantics to improve lesion delineation, yet many existing models bind cross-modal fusion, supervision, and decoder design into a task-specific architecture.

By Yungeng Liu, Xuanzi Fang, Haijin Zeng, Qi Dai, Yongyong Chen

llms computer-vision multimodal safety

arXiv AI

Jul 13

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

arXiv:2607. 08964v1 Announce Type: new Abstract: AI agents have become capable of autonomously completing short, well-specified tasks.

By Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, LeoweiLiang

agents multimodal benchmarks

arXiv Machine Learning

Jul 13

TSAI-MetaFraud: A Benchmark Dataset for Financial Fraud Transaction and Behavioral Risk Detection in Metaverse Ecosystems

arXiv:2607. 09528v1 Announce Type: new Abstract: The emergence of metaverse platforms has created virtual economies that introduce new challenges related to fraud, bot activity, and illicit financial behavior.

By Refat Ishrak Hemel, Ehsan Hallaji, Roozbeh Razavi-Far

multimodal benchmarks

arXiv AI

Jul 13

GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs

arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.

By Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal

llms fine-tuning multimodal safety

arXiv AI

Jul 13

Prompt-Driven Exploration

arXiv:2607. 08837v1 Announce Type: cross Abstract: Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers.

By Sunshine Jiang, John Marangola, David Zhang, Raghuram Kowdeed, Ruiyang Luo, Nitish Dashora, Richard Li, Pulkit Agrawal, Zhang-Wei Hong

llms reinforcement-learning robotics multimodal

arXiv Machine Learning

Jul 13

TSRouter: Dynamic Modality-Model Selection for Time Series Reasoning

arXiv:2607. 08940v1 Announce Type: new Abstract: Time series reasoning is essential for real-world problem-solving.

By Fangxu Yu, Tao Feng, Dehai Min, Lu Cheng, Ge Liu, Tianyi Zhou

llms multimodal

arXiv AI

Jul 13

ReGen: Hierarchical Multi-Prompt Representation Generation for Efficient Waveform Diffusion Models

arXiv:2607. 09134v1 Announce Type: cross Abstract: Representation alignment (REPA) has been investigated to accelerate diffusion training, but we observe that regularizing intermediate representations in diffusion Transformers (DiT) may implicitly entangle latents and limit generative capacity.

By Sang-Hoon Lee, Ha-Yeong Choi

llms diffusion multimodal safety

arXiv AI

Jul 13

Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

arXiv:2607. 09438v1 Announce Type: cross Abstract: Test-time scaling (TTS) reliably improves reasoning in large language models, but whether it transfers to small open vision-language models remains unclear.

By Spiros Baxevanakis, Peng-Jian Yang

llms multimodal benchmarks

arXiv AI

Jul 13

ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts

arXiv:2607. 09526v1 Announce Type: cross Abstract: Foundation models are reshaping computational pathology, yet their capabilities remain shaped by pretraining objectives, data sources, and spatial scales, fragmenting complementary expertise across separate backbones.

By Jiawen Li, Tian Guan, Huijuan Shi, Xitong Ling, Mingxi Fu, Anjia Han, Chao He, Yonghong He

efficiency multimodal

arXiv Machine Learning

Jul 13

Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

arXiv:2607. 08839v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are typically designed under the assumption that all modalities available during training will also be accessible at inference.

By Dominick Reilly, Qiyu Wu, Hiromi Wakaki, Srijan Das, Yuki Mistufuji

llms multimodal safety

arXiv AI

Jul 13

TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models

arXiv:2607. 09562v1 Announce Type: cross Abstract: Medical Vision-Language Models (VLMs) exhibit strong zero-shot performance, yet their effectiveness still declines on out-of-distribution (OOD) data due to domain shifts and class bias inherited from large-scale pretraining.

By Tianyou Jiang, Ziyu Zhou

llms multimodal safety

arXiv Machine Learning

Jul 13

Learning More from Less: Reinforcement Learning from Hindsight

arXiv:2607. 09042v1 Announce Type: new Abstract: Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern.

By Iris Xu, Sunshine Jiang, John Marangola, Nitish Dashora, Richard Li, Thomas Liu, Zexue He, Yuheng Zhi, Alex Pentland, Pulkit Agrawal, Zhang-Wei Hong

llms reinforcement-learning robotics multimodal

Get the daily briefing