Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

3,844 stories · RSS feed

arXiv AI

Aug 13

P2MFDS: A Privacy-Preserving Multimodal Fall Detection System for Elderly People in Bathroom Environments

arXiv:2506. 17332v2 Announce Type: replace-cross Abstract: By 2050, people aged 65 and over are projected to make up 16% of the global population.

By Haitian Wang, Yiren Wang, Xinyu Wang, Yumeng Miao, Yuliang Zhang, Yu Zhang, Atif Mansoor

multimodal benchmarks safety

arXiv AI

Aug 13

How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?

arXiv:2603. 00056v2 Announce Type: replace-cross Abstract: STEM Mental models can play a critical role in assessing students' conceptual understanding of a topic.

By Pritam Sil, Durgaprasad Karnam, Vinay Reddy Venumuddala, Pushpak Bhattacharyya

multimodal

arXiv AI

Aug 13

Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework

arXiv:2608. 11891v1 Announce Type: cross Abstract: Governments increasingly fund indigenous foundation models to strengthen national AI capability, digital sovereignty, and multilingual computing.

By Avinash Agarwal, Vridhi Jain

agents multimodal benchmarks

arXiv AI

Aug 13

Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence

arXiv:2608. 12290v1 Announce Type: cross Abstract: Modern black-box Image-to-Video (I2V) models offer powerful capabilities in automated content creation, yet their lack of fine-grained control and reliability presents significant challenges in professional workflows.

By Aman Tyagi, Hemanth Boinpally, Jonathan Chen, Douglas Gebert, Steven Hickson

llms agents diffusion multimodal benchmarks

arXiv Machine Learning

Aug 13

Click2Poly: A VLM for vector mapping buildings and walls

arXiv:2608. 11424v1 Announce Type: new Abstract: Accurate vector mapping of buildings and walls is critical for geospatial applications but remains a labor-intensive process.

By Nicolas Girard, Jawher Ben Abdallah, Arno Gobbin, Liuyun Duan, Sacha Lepretre

llms multimodal

arXiv AI

Aug 13

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation

arXiv:2608. 11681v1 Announce Type: cross Abstract: This work addresses the challenge of open-vocabulary instance segmentation (OVIS) and open-set panoptic segmentation (OSPS), which aim to recognize both predefined and unseen object categories without exhaustive human annotations.

By Duy Tran Thanh, Yeejin Lee, Byeongkeun Kang

llms computer-vision multimodal benchmarks safety

arXiv AI

Aug 13

G0.5: One Autoregressive Stream for Robot Reasoning and Action

arXiv:2608. 11739v1 Announce Type: cross Abstract: The prevailing recipe for Vision-Language-Action (VLA) models couples a pretrained VLM with a separately trained flow-matching action expert.

By Yicheng Liu, Zibin Dong, Baijun Ye, Tianyuan Yuan, Tao Jiang, Anqi Yang, Shicheng Cao, Haonan Liu, Yue Sun, Zihan Guo, Xiao Liu, Dong Ke, Changxun Pan, Chenru Wu, Tailai Cheng, Xiaoshu Ren, Xinlei Zhang, Jianning Cui, Zijie Zhao, Haoyu Zhang, Kaiming Xu, Haodong Yang, Bowen Zhang, Jiahui Niu, Shaoting Zhu, Shiduo Zhang, Hang Zhao

llms nlp robotics fine-tuning multimodal benchmarks

arXiv AI

Aug 13

JieZi: A Large-Scale Expert-Audited Dataset and Benchmark for Ancient Chinese Character Exegesis

arXiv:2608. 11741v1 Announce Type: cross Abstract: The scholarly exegesis of ancient Chinese characters demands integrating visual observation, linguistic analysis, and historical context.

By Ran Li, Huiguo He, Jiahuan Cao, Junle Liu, Hiuyi Cheng, Lianwen Jin

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Aug 13

Inverse Theory of Mind Modeling for Content Recommendation: From Web Browsing to Dynamic Intelligent Interfaces

arXiv:2608. 11354v1 Announce Type: new Abstract: Modern recommender systems treat observed actions as reliable proxies for user preferences, yet interactions often reflect exploration or comparison rather than stable preference expression.

By Mengyu Chen, Feiyu Lu, Chun-Fu Chen, Lucas Vinh Tran, Jay Katukuri

llms multimodal safety

arXiv AI

Aug 13

Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System

arXiv:2608. 11738v1 Announce Type: cross Abstract: Multimodal Large Language Model (MLLM)-based UAV aerial image understanding and reasoning is essential for aerial intelligence yet poses distinct challenges arising from extreme scale variation, arbitrary camera orientations, and high object density.

By Haoyu Zhang, Shuoxun Zhang, Peng Ye, Lin Zhang, Jiakang Yuan, Shenghong Yi, Yuening Wang, Tao Chen

llms agents multimodal benchmarks

arXiv AI

Aug 13

A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

arXiv:2603. 11211v3 Announce Type: replace-cross Abstract: Incremental Learning (IL) aims to learn new tasks while preserving previously acquired knowledge.

By Haihua Luo, Xuming Ran, Jiangrong Shen, Timo H\"am\"al\"ainen, Zhonghua Chen, Qi Xu, Fengyu Cong

llms multimodal

arXiv AI

Aug 13

FM-LLM: A frequency-enhanced mixture-of-experts framework for adapting LLMs to time series forecasting

arXiv:2608. 11623v1 Announce Type: cross Abstract: Recent advances in Large Language Models (LLMs) have spurred cross-modal solutions for time-series forecasting.

By Rentao Gu, Yihang Ding, Junjie Li, Yi Ding, Weijing Sang, Xiaoli Huo, Xin Qin, Yuefeng Ji

llms multimodal benchmarks safety

arXiv AI

Aug 13

How China-Origin Vision-Language Models Move from Refusal to Reframing in State Alignment

arXiv:2608. 11816v1 Announce Type: cross Abstract: State-aligned distortion has been documented in China-origin text-based large language models (LLMs), but whether, and in what form, it arises in multimodal systems has not been systematically examined.

By Guang Yang, Fengchen Liu, Alex Wang, Homa Hosseinmardi, Amir Ghasemian

llms multimodal benchmarks safety

arXiv AI

Aug 13

SportD: How do VLMs physically strategize?

arXiv:2607. 14616v3 Announce Type: replace Abstract: Vision-language models (VLMs) can describe a scene, but can they act well within one?

By Jasin Cekinmez, Addison J. Wu, Haotian Xia, Kyumin Andrew Shim, Anay Putty, Jinglin Xiao, Zhuohan Liu, Leo Liu, Weining Shen

llms multimodal

arXiv AI

Aug 13

SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward

arXiv:2608. 12220v1 Announce Type: cross Abstract: Existing Vision-Language Models (VLMs) exhibits a critical bottleneck in robust spatial reasoning.

By Zile Zhou, Huining Yuan, Weichen Zhang, Xinlei Chen, Xiao-ping Zhang

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Aug 13

Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models

arXiv:2605. 16409v3 Announce Type: replace-cross Abstract: Optical character recognition (OCR) and multilingual scene-text understanding remain challenging for multimodal large language models (MLLMs), particularly in real-world images containing small or degraded text, cluttered layouts, occlusion, handwriting, and complex typography.

By Qinwu Xu, Yifan Jiang, Haoyu Ren

llms computer-vision fine-tuning multimodal benchmarks safety

arXiv AI

Aug 13

DreamFly: Causal Memory and Receding-Horizon Diffusion Planning for Aerial Vision-Language Navigation

arXiv:2608. 12308v1 Announce Type: cross Abstract: Aerial vision-language navigation (VLN) requires an embodied agent to integrate visual evidence over time, plan future actions, and determine when it has reached a navigation goal under partial observability.

By Yan Deng, Fei Xu

agents diffusion robotics multimodal benchmarks

arXiv AI

Aug 13

VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

arXiv:2603. 17450v2 Announce Type: replace-cross Abstract: Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations.

By Junyoung Kim, Woojoo Kim, Wonbin Kweon, Jaehyung Lim, Dongha Kim, Hwanjo Yu

llms rag fine-tuning multimodal

arXiv AI

Aug 13

CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D

arXiv:2509. 24528v4 Announce Type: replace-cross Abstract: Object retrieval from a scene has become a new trend of research due to its numerous applications.

By Mohamad Amin Mirzaei, Pantea Amoie, Ali Ekhterachian, Matin Mirzababaei, Babak Khalaj

llms rag computer-vision multimodal benchmarks

arXiv AI

Aug 13

Dual-Domain Cross-Modal Decoding for Clinical Text-Guided Medical Image Segmentation

arXiv:2608. 11335v1 Announce Type: cross Abstract: Clinical text can narrow down what to segment, but recent text-guided designs emphasize spatial alignment while overlooking frequency content that governs texture and boundaries.

By Md Maklachur Rahman, Tracy Hammond

computer-vision multimodal safety