Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,972 stories · RSS feed

arXiv AI

Jul 7

Parameter Efficient Multimodal Instruction Tuning for Romanian Vision Language Models

arXiv:2512. 14926v2 Announce Type: replace-cross Abstract: Focusing on low-resource languages is an essential step toward democratizing generative AI.

By George-Andrei Dima, R\u{a}zvan-Alexandru Sm\u{a}du, Dumitru-Clementin Cercel

llms nlp fine-tuning multimodal

arXiv Machine Learning

Jul 7

Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speech

arXiv:2607. 05165v1 Announce Type: new Abstract: Non-invasive brain-to-speech decoding aims to restore communication to patients suffering from neurodegenerative disease, without the risks of neurosurgery.

By Benjamin Ballyk, Teyun Kwon, Miran \"Ozdogan, Oiwi Parker Jones

multimodal

arXiv AI

Jul 7

MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understanding

arXiv:2607. 02934v1 Announce Type: cross Abstract: Materials phase diagrams are a core knowledge representation in materials science, encoding temperature,composition, phase stability, and phase transformation pathways, with their full understanding requiring thermodynamic mechanism analysis and scientific reasoning.

By Hanwen Wang, Sihan Liang, Zhiwei Liu, Yangang Wang, Wei Yan, Yuqin Liu, Zongguo Wang

multimodal benchmarks safety

arXiv AI

Jul 7

Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

arXiv:2607. 03748v1 Announce Type: new Abstract: Unified multi-modal models (UMMs) have shown promising interleaved text-image reasoning capabilities, yet effectively optimizing such multi-turn generation via reinforcement learning (RL) remains an open challenge.

By Zican Hu, Xuyang Hu, Yiming Liu, Zuwei Long, Wei Liu, Yunzhuo Hao, Jiawei Gu, Linjie Li, Yu Cheng, Zhenhong Sun, Weibo Gu, Xing Sun, Zhi Wang

llms diffusion reinforcement-learning multimodal benchmarks

arXiv AI

Jul 7

Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

arXiv:2607. 03233v1 Announce Type: cross Abstract: The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation.

By Eduardo Almeida Palmieri, Mohamed Chahine Ghanem, Dipo Dunsin, Zubair Baig, Ed de Quincey, Kim-Kwang Raymond Choo

llms rag agents nlp fine-tuning multimodal benchmarks safety

arXiv AI

Jul 7

GeoSelect: Spatial-Program Execution for Training-Free Referring Remote Sensing Image Segmentation

arXiv:2607. 03869v1 Announce Type: cross Abstract: Referring remote sensing image segmentation isolates the object named by a natural-language expression in an aerial image.

By Yuhang Jiang, Guohui Deng, Miaozhong Xu, Chao Ruan, Jinling Zhao, Linsheng Huang

llms computer-vision multimodal

arXiv AI

Jul 7

VLA Grounder: Language-Conditioning Space Optimization for Black-Box VLA Models

arXiv:2607. 04517v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are commonly treated as end-to-end action policies conditioned on natural-language task descriptions.

By Damir Shodiev, Aleksei Staroverov, Nikita Kachaev, Alexey K. Kovalev, Aleksandr I. Panov

reinforcement-learning robotics multimodal

arXiv Machine Learning

Jul 7

OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction

arXiv:2604. 16878v2 Announce Type: replace Abstract: Early prediction of severe clinical deterioration and remaining length of stay can enable timely intervention and better resource allocation in high-acuity settings such as the ICU.

By Zhongyuan Liang, Junhyung Jo, Hyang-Jung Lee, Sang Kyu Kim, Irene Y. Chen

fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Jul 7

Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions

arXiv:2604. 11730v4 Announce Type: replace-cross Abstract: Using behavioural science, health interventions focus on behaviour change by providing a framework to help patients acquire and maintain healthy habits that improve medical outcomes.

By Manuela Gonz\'alez-Gonz\'alez, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Lorenzo Sia, Nicolas Richet, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L Bacon, Eric Granger

llms fine-tuning multimodal

arXiv Machine Learning

Jul 7

H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

arXiv:2607. 02592v1 Announce Type: cross Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm by providing supervision on student-generated trajectories.

By Qixiang Yin, Huanjin Yao, Yuchen Cai, Jianghao Chen, Ziyi Wang, Min Yang, Fei Su, Zhicheng Zhao

efficiency multimodal benchmarks

arXiv AI

Jul 7

High-Fidelity One-Step Generative Visuomotor Policy via Recursive Correction, Frequency Consistency, and Contrastive Flow Matching

arXiv:2607. 03865v1 Announce Type: cross Abstract: Generative models such as diffusion and flow matching have advanced robotic visuomotor policies by modeling multimodal action distributions, but their multi-step sampling or ODE solving introduces inference latency.

By Yuran Chen, Xinye Cai, Zhonglin Gong, Yang Huang

diffusion robotics efficiency multimodal

arXiv AI

Jul 7

BanglaMemeEvidence: A Multimodal Benchmark Dataset for Explanatory Evidence Detection in Bengali Memes

arXiv:2607. 03981v1 Announce Type: cross Abstract: Memes have become influential communication tools on social media, combining viral visuals with concise messaging to convey impactful ideas.

By Fatema Tuj Johora Faria, Mukaffi Bin Moin, Md. Mahfuzur Rahman, Pronay Debnath, Asif Iftekher Fahim, Faisal Muhammad Shah

computer-vision nlp multimodal benchmarks

arXiv AI

Jul 7

EGRA:Toward Enhanced Behavior Graphs and Representation Alignment for Multimodal Recommendation

arXiv:2508. 16170v2 Announce Type: replace-cross Abstract: MultiModal Recommendation (MMR) systems have emerged as a promising solution for improving recommendation quality by leveraging rich item-side modality information, prompting a surge of diverse methods.

By Xiaoxiong Zhang, Xin Zhou, Zhiwei Zeng, Yongjie Wang, Zhiqi Shen

llms efficiency multimodal safety

arXiv AI

Jul 7

SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

arXiv:2607. 05365v1 Announce Type: cross Abstract: Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech.

By Thomas Thebaud, Yuzhe Wang, Hao Zhang, Sathvik Manikantan Napa Ugandhar, Ashish Hallur, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Velazquez

llms multimodal benchmarks

arXiv AI

Jul 7

SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards

arXiv:2511. 07403v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language tasks, but continue to struggle with spatial reasoning.

By Hunar Batra, Haoqin Tu, Hardy Chen, Yuanze Lin, Cihang Xie, Ronald Clark

llms reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jul 7

Generative Semantic Multi-Object Tracking: A Large-Scale Benchmark and an MLLM-Driven Reasoning Framework

arXiv:2601. 06550v3 Announce Type: replace-cross Abstract: Semantic Multi-Object Tracking (SMOT) is evolving from purely geometric localization toward comprehensive video understanding.

By Pan Liao, Feng Yang, Di Wu, Jinwen Yu, Wang Zhao, Dingwen Zhang

llms multimodal benchmarks safety

arXiv AI

Jul 7

From Fixed to Free Cameras: Calibration-Free View-Robust Vision-Language-Action Model

arXiv:2607. 05396v1 Announce Type: cross Abstract: Real-world robot deployment rarely maintains the training-stage camera setup, where cameras often experience repositioning or remounting depending on actual scenarios.

By Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Shijian Lu, Gongjie Zhang, Ran Xu

robotics multimodal

arXiv AI

Jul 7

Echoes of Unrest: A Multimodal NLP Framework for Early Warning of Fake News and Violence-Driven Mob Activity

arXiv:2607. 02734v1 Announce Type: cross Abstract: Rapid growth in social media has transformed global communication by enabling fast information exchange, but it has also accelerated the spread of misinformation.

By Md. Maruf Bangabashi, Tahmid Hasan, Golam Mahmud, Md. Mostafijur Rahman, Md. Toufiqur Rahman, Jahanur Biswas

rag nlp multimodal benchmarks

arXiv AI

Jul 7

DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics

arXiv:2607. 04112v1 Announce Type: cross Abstract: Multimodal LLMs struggle to systematically model the temporal evolution of visual scenes in videos or multi-image sequences.

By Silin Gao, Hao Zhao, Zeming Chen, Sepideh Mamooler, Antara Raaghavi Bhattacharya, Qiyu Wu, Hiromi Wakaki, Yuki Mitsufuji, Li Mi, Syrielle Montariol, Antoine Bosselut

llms multimodal

arXiv AI

Jul 7

CRISP: A Spatiotemporal Camera-Radar Backbone for Driving via Forecasting-Based World-Model Pretraining

arXiv:2607. 04541v1 Announce Type: cross Abstract: Camera-radar (CR) fusion is a practical sensing configuration for autonomous driving, but existing models are typically trained with task-specific supervision, limiting reusable representation learning.

By Jingyu Song, Yi Liu, Katherine A. Skinner

agents robotics multimodal

Get the daily briefing