Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

3,891 stories · RSS feed

arXiv Machine Learning

Aug 10

Neurai-VN Benchmark: Standardized Machine Learning Models for Multimodal Digital Phenotyping in Mental Health Classification

arXiv:2607. 25232v2 Announce Type: replace Abstract: Digital phenotyping (DP) using smartphones and wearable devices has shown considerable potential for mental health monitoring.

By Quoc-Cuong Pham, Hoang-Thuy-Duong Vu, Thi-Thanh-Huong Ha, Huy-Hieu Pham

multimodal benchmarks

arXiv Machine Learning

Aug 10

TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution

arXiv:2603. 27723v2 Announce Type: replace Abstract: Multimodal-attributed graphs (MAGs) are a fundamental data structure for multimodal graph learning (MGL), enabling both graph-centric and modality-centric tasks.

By Yinlin Zhu, Xunkai Li, Di Wu, Wang Luo, Miao Hu, Guocong Quan

rag multimodal benchmarks safety

arXiv AI

Aug 10

MetaSICL: Globalizing Auditory LLMs for Underserved Speakers and Languages via Meta Speech In-Context Learning

arXiv:2601. 18904v3 Announce Type: replace-cross Abstract: Generative AI for speech and audio is increasingly expected to serve users across languages, cultures, and communities, yet current auditory Large Language Models (LLMs) are still largely trained and evaluated on high-resource data.

By Haolong Zheng, Siyin Wang, Zengrui Jin, Mark Hasegawa-Johnson

llms reinforcement-learning fine-tuning multimodal

arXiv AI

Aug 10

CertBind from Multimodal Connectivity to Certifiable Retrieval Decisions

arXiv:2608. 06516v1 Announce Type: cross Abstract: Lightweight connectors make frozen multimodal encoders composable at the representation level.

By Shuheng Cao, Zhenhao Zhang, Ruiqi Chen, Renjie Cao, Weijia Zhang, Siyu Zhang, Jiaxin Liu, Xiangyu Zeng, Haotian Geng, Fan Gu

multimodal

arXiv AI

Aug 10

Diffusion LLMs as Targets and Adversaries: Mechanistic Safety Exploits

arXiv:2608. 07430v1 Announce Type: cross Abstract: Diffusion Large Language Models (DLLMs) replace autoregressive next-token prediction with iterative parallel denoising, yet their internal safety mechanisms remain poorly understood.

By Elena Dumitrescu, Gert Lek, Lydia Y. Chen, J\'er\'emie Decouchant

llms diffusion efficiency multimodal safety

arXiv Machine Learning

Aug 10

The Sparsity Whisperer

arXiv:2608. 06630v1 Announce Type: new Abstract: Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs.

By Linghao Kong, Inimai Subramanian, Micah Adler, Dan Alistarh, Dan Gutfreund, Nir Shavit

llms efficiency multimodal

arXiv AI

Aug 10

CASA: Classification Augmented with Safety Attention for Robust Multimodal Alignment

arXiv:2604. 00310v2 Announce Type: replace-cross Abstract: Multimodal large-language models (MLLMs) often experience degraded safety alignment when harmful queries exploit cross-modal interactions.

By Anurag Kumar, Raghuveer Peri, Jon Burnsky, Alexandru Nelus, Rohit Paturi, Srikanth Vishnubhotla, Yanjun Qi

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Aug 10

LifelongCrossNav: Persistent 3D Semantic Memory for Cross-Floor Multi-Object Navigation

arXiv:2608. 07079v1 Announce Type: cross Abstract: Object-goal navigation has made substantial progress in semantic perception and exploration, yet persistent memory for multi-object navigation and cross-floor navigation are still commonly addressed separately.

By Zehui Li, Zihao Sun, Jiawei Xu, Zheqi He, Xiaoqiang Zhang, Jing-Shu Zheng, Lu Liu, Dahui Gao, Xiuwan Chen

agents multimodal benchmarks

arXiv AI

Aug 10

Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery

arXiv:2608. 06931v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly involved in scientific discovery, yet it remains unclear whether they can support complex real laboratory science.

By Taolin Han, Yuchen Zhang, Jinghang Wang, Yun Wu, Wai Yuet Chiu, Zhaohai Li, Yifei Zhang, Jinxin Wang, Yuhao Zhou, Chen Zhao, Jiajia Li, Jiaxin Li, Qile Jin, Kewei Sun, Shuang Wu, Weiqi Zhai, Renquan Lv, Junchao Li, Ruodan Chen, Qingteng Chen, Zhibo Yang, Hu Wei, Lin Qu, Shuai Bai, Bing Zhao

llms agents multimodal benchmarks

arXiv Machine Learning

Aug 10

AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agents

arXiv:2606. 05597v3 Announce Type: replace Abstract: Training vision-language web agents with multi-step RL is compute-intensive, with two dominant forms of inefficiency: idle GPUs in synchronous RL, and trajectories that use more steps and tokens than necessary.

By Hao Bai, Rui Yang, Chenlu Ye, Spencer Whitehead, Aviral Kumar, Tong Zhang

agents reinforcement-learning efficiency multimodal

arXiv AI

Aug 10

Surg-UniWorld: A Unified Surgical World Model with Multimodal Control Experts

arXiv:2608. 06770v1 Announce Type: new Abstract: Controllable surgical world models can provide a generative foundation for surgical artificial intelligence and simulation by synthesizing realistic instrument--tissue interactions.

By Rulin Zhou, Wanhao Liu, Guoheng Ma, Liangjin Shao, Qiujie Song, Yidu Wang, Guankun Wang, Tong Chen, Long Bai, Luping Zhou, Hongliang Ren

diffusion multimodal benchmarks

arXiv AI

Aug 10

AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Models

arXiv:2608. 06699v1 Announce Type: new Abstract: Agentic multimodal large language models (MLLMs) extend multimodal perception and reasoning with planning, tool use, and interaction in dynamic environments.

By Zibo Shao, Baochen Xiong, Chengdong Xu, Linhui Xiao, Kaichen Li, Haoran Gong, Yan Li, Yaguang Song, Xiaoshan Yang

llms agents multimodal benchmarks

arXiv AI

Aug 10

FedVAR: Prototype-Aligned Federated Framework for Video Anomaly Recognition

arXiv:2608. 06876v1 Announce Type: cross Abstract: In the era of Industrial Internet of Things (IIoT) and Cyber-Physical Systems (CPS), Federated Learning (FL) offers a promising decentralized intelligence paradigm for Video Anomaly Recognition (VAR).

By Ghani Haider, Majid Kundroo, Boyun Eom, Dong Hwan Park, Chen Chen, Taehong Kim

llms multimodal benchmarks safety

arXiv AI

Aug 10

Judge a Book by its Cover: Investigating Multi-Modal LLMs for Multi-Page Handwritten Document Transcription

arXiv:2502. 20295v3 Announce Type: replace-cross Abstract: Handwriting text recognition (HTR) remains a challenging task.

By Benjamin Gutteridge, Matthew Thomas Jackson, Toni Kukurin, Xiaowen Dong

llms computer-vision fine-tuning multimodal benchmarks

arXiv AI

Aug 10

RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs

arXiv:2608. 07088v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) encode images as long visual token sequences, making prefilling and KV-cache storage expensive.

By Qiyanhui Lu, Han Wu, Rongjian Xu, Tingzhang Luo, Cheng Fan, Xinghao Chen, Minjing Dong, Jufeng Yang, Jianyuan Guo

llms efficiency multimodal

arXiv AI

Aug 10

DATAREEL: Automated Data-Driven Video Story Generation with Animations

arXiv:2604. 25220v2 Announce Type: replace Abstract: Data videos combine animated visualizations with synchronized narration to communicate quantitative information and are widely used in journalism, education, and public communication.

By Ridwan Mahbub, Syem Aziz, Mizanur Rahman, Mahir Ahmed, Shadikur Rahman, Shafiq Joty, Enamul Hoque

llms agents nlp multimodal benchmarks

arXiv AI

Aug 10

Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

arXiv:2608. 06411v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens.

By Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

llms efficiency multimodal benchmarks

arXiv AI

Aug 10

WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader

arXiv:2608. 06474v1 Announce Type: new Abstract: Large language models increasingly generate complete websites from natural-language descriptions, and reinforcement learning has become a central approach to closing their remaining functional gap.

By Boshui Chen, Huiping Liu, Shaolei Zhang

llms agents reinforcement-learning multimodal

arXiv AI

Aug 10

TransSLR: A Lightweight Transformer for Sign Language Recognition

arXiv:2608. 06407v1 Announce Type: cross Abstract: Automated Sign Language Recognition for under-represented languages remains a largely unsolved problem.

By Lucia Yen Wanchi, Samuel Johnny, Victor Tolulope Olufemi, Emmanuel Aaron, Moise Busogi

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 10

Sampling via Stochastic Interpolants by Langevin-based Velocity and Initialization Estimation in Flow ODEs

arXiv:2601. 08527v3 Announce Type: replace-cross Abstract: We propose a novel method for sampling from unnormalized Boltzmann densities based on a probability flow ordinary differential equation (ODE) derived from linear stochastic interpolants.

By Chenguang Duan, Yuling Jiao, Gabriele Steidl, Christian Wald, Jerry Zhijian Yang, Ruizhe Zhang

multimodal