Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,631 stories · RSS feed

arXiv Machine Learning

Jun 2

PaCX-MAE: Physiology-Augmented Chest X-Ray Masked Autoencoder

arXiv:2606. 01537v1 Announce Type: cross Abstract: Clinical diagnosis often requires combining imaging with physiological measurements, yet deployed models typically operate on unimodal data.

By Yancheng Liu, Kenichi Maeda, Manan Pancholy

rag computer-vision efficiency multimodal benchmarks

arXiv Machine Learning

Jun 2

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

arXiv:2606. 02576v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential.

By Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang, Da-Wei Zhou

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 2

CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

arXiv:2606. 00931v1 Announce Type: cross Abstract: Instruction-guided image editing is becoming a general interface for visual work, yet existing benchmarks still focus largely on narrow appearance edits and do not fully capture the diversity of real-image tasks in professional workflows.

By Fangzhou Lin, Peiran Li, Lingyu Xu, Wenjing Chen, Qianwen Ge, Shuo Xing, Mingyang Wu, Xiangbo Gao, Siyuan Yang, Kazunori Yamada, Ziming Zhang, Haichong Zhang, Zhen Dong, Ming-Hsuan Yang, Zhengzhong Tu

agents computer-vision multimodal benchmarks

arXiv AI

Jun 2

CARES: Context-Aware Resolution Selector for VLMs

arXiv:2510. 19496v3 Announce Type: replace-cross Abstract: Large vision-language models (VLMs) commonly process images at native or high resolution to remain effective across tasks.

By Moshe Kimhi, Nimrod Shabtay, Raja Giryes, Chaim Baskin, Eli Schwartz

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

Longitudinal Multimodal Sensing of Physical Activity and Well-Being in Older Adults

arXiv:2606. 00345v1 Announce Type: new Abstract: Wearable and mobile sensing technologies enable continuous monitoring of human behavior and health in real-world settings.

By Flavio Di Martino, Mattia G. Campana, Marcello Magno, Lorenza Pratali, Franca Delmastro

multimodal safety

arXiv Machine Learning

Jun 2

Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning

arXiv:2606. 00851v1 Announce Type: cross Abstract: Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately, yet everyday speech often carries weak, neutral, or ambiguous affective cues.

By Sukru Samet Dindar, Riki Shimizu, Xilin Jiang, Nima Mesgarani

multimodal safety

arXiv AI

Jun 2

MURMUR: An Efficient Inference System for Long-Form ASR

arXiv:2606. 01483v1 Announce Type: cross Abstract: Long-form automatic speech recognition (ASR) requires both high accuracy and low latency, but existing systems force a trade-off between the two.

By Wei-Tzu Lee, Keisuke Kamahori, Baris Kasikci

efficiency multimodal

arXiv Machine Learning

Jun 2

Cross-Generational Transfer of Adversarial Attacks Reveals Non-Monotonic Safety Alignment in LLMs

arXiv:2606. 00813v1 Announce Type: cross Abstract: Safety alignment in LLMs does not improve monotonically across model generations.

By Subhadip Mitra

llms multimodal benchmarks safety

arXiv AI

Jun 2

ES-Merging: Biological MLLM Merging via Embedding Space Signals

arXiv:2603. 14405v2 Announce Type: replace-cross Abstract: Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery.

By Wonbin Lee, Dongki Kim, Sung Ju Hwang

llms rag multimodal

arXiv Machine Learning

Jun 2

Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

arXiv:2606. 01710v1 Announce Type: cross Abstract: Vision-Language models (VLMs), such as CLIP, achieve powerful zero-shot classification.

By Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie, Sarah Erfani

llms rag fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions

arXiv:2606. 01703v1 Announce Type: cross Abstract: We address the challenge of generating high-fidelity, long-form soundtracks that remain coherent across scene transitions.

By Jiashuo Yu, Yao Yao, Boyu Chen, Alex Wang

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 2

IstGPT: LLM-based Anomaly Detection for Spatial-Temporal Graph in Industrial Systems

arXiv:2606. 01691v1 Announce Type: cross Abstract: Industrial Internet systems face increasing threats from sophisticated industrial control system (ICS) attacks, resulting in critical safety incidents.

By Yuchen Zhang, Ning Xi, Pengbin Feng, Shigang Liu, Jianfeng Ma, Yulong Shen, Yanan Sun, Xiaolin Zhou

llms robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

GLENS: Global Search via Learning from Solver Iterates with Diffusion Models

arXiv:2606. 00366v1 Announce Type: new Abstract: We consider the problem of generating a large collection of initial guesses for local minima of multimodal non-convex continuous optimization problems.

By Anjian Li, Bartolomeo Stellato, Ryne Beeson

diffusion robotics multimodal benchmarks

arXiv AI

Jun 2

Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibration

arXiv:2606. 00145v1 Announce Type: cross Abstract: Vision-language-action (VLA) agents can execute natural-language instructions, yet deployed systems still lack an operational interface: deciding when the instruction is complete.

By Yusuke Sano, Takeshi Itoga

agents multimodal benchmarks

arXiv AI

Jun 2

StreamingVLM: Real-Time Understanding for Infinite Video Streams

arXiv:2510. 09608v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage.

By Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Yao Lu, Song Han

llms agents fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Jun 2

StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning

arXiv:2606. 00148v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) often know the rule but pick the wrong answer: on abstract visual reasoning (AVR) tasks, a model can describe what it sees and name the underlying pattern, yet still fail to choose the matching candidate.

By Xixiang He, Baiqi Wu, Xingming Li, Ao Cheng, Qiyao Sun, Xuanyu Ji, Qingyong Hu

llms multimodal benchmarks

arXiv AI

Jun 2

Silent Failures in Physical AI: A Literature Review of Runtime Action Authorization for Autonomous Systems

arXiv:2606. 00090v1 Announce Type: cross Abstract: Physical AI systems increasingly map multimodal observations, language instructions, and learned world representations into physically consequential actions.

By Barak Or

agents robotics multimodal benchmarks safety

arXiv AI

Jun 2

Planktonzilla: Multimodal dataset and models for understanding plankton ecosystems

arXiv:2606. 00080v1 Announce Type: cross Abstract: Marine plankton underpin aquatic food webs and play a key role in global CO2 sequestration, making reliable species identification critical for understanding ocean health and climate feedbacks.

By Alan Gerson Contreras Montanares, Luis Valenzuela, Luis Mart\'i, Nayat Sanchez-Pi

multimodal

arXiv AI

Jun 2

Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning

arXiv:2503. 11832v5 Announce Type: replace Abstract: Recent vision language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs, particularly text and images.

By Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, Sijia Liu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

When Jokes Cross the Line: Analyzing Regular Humor and Dark Humor in YouTube Shorts

arXiv:2606. 00046v1 Announce Type: cross Abstract: Video platforms such as YouTube have reshaped how users engage with entertainment and information, emphasizing brief, highly engaging content such as Shorts.

By Sydney Johns, Sanjeev Parthasarathy, Shantnu Bhalla, Vaibhav Garg

llms multimodal benchmarks

Get the daily briefing