Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

3,800 stories · RSS feed

arXiv AI

1d ago

VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

arXiv:2608. 15265v1 Announce Type: new Abstract: Constructing an interactive 3D open world from a user query is important.

By Yansong Ning, Jingwen Ye, Zhongkai Wu, Yang Sun, Yiqin Zhu, Xingyi Li, Weidong Zhang, Hao Liu

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

1d ago

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

arXiv:2608. 14841v1 Announce Type: new Abstract: Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines.

By Guanchen Wu, Jiayuan Ding, Subhabrata Mukherjee, Carl Yang

llms nlp multimodal

arXiv AI

1d ago

Toward AI-Friendly Cartography: Understanding How Color Design Influences Foundation Model Spatial Reasoning on Sequential Choropleth Maps

arXiv:2608. 15736v1 Announce Type: new Abstract: Foundation models (FMs) increasingly support multimodal and geospatial reasoning, yet it remains unclear whether cartographic principles designed for human perception are equally effective for machines.

By Yonghe Sun, Zhenjia Liu, Hua Liao, Wenjia Xu, Nai Yang, Weihua Dong, Zhiwei Wei

fine-tuning multimodal benchmarks

arXiv AI

1d ago

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

arXiv:2608. 15930v1 Announce Type: new Abstract: Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution.

By Zihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke, Mengcheng Lan, Weixian Lei, Hanxuan Li, Honglin Li, Xiyun Li, Zaitang Li, Leowei Liang, Xin Luo, Haozhe Ma, Jiayi Mao, Zhoujie Pan, Can Qin, Tianyuan Qu, Weiqi Wang, Wenkai Wang, Yonglin Wang, Yuxin Wang, Chenxu Wu, Yingchen Yu, Chenyu Zhang, Yuhao Zheng

agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

1d ago

TRACE: Trajectory Aware Reasoning for Multi-Turn Adversarial Conversation Evaluation

arXiv:2608. 15594v1 Announce Type: new Abstract: Multi-turn jailbreak attacks have emerged as a critical safety threat to LLMs, as harmful objectives are decomposed across a sequence of apparently benign turns to bypass guardrails.

By Md Messal Monem Miah, Adrita Anika, Zhiyuan Yu, Ruihong Huang

llms robotics fine-tuning multimodal benchmarks safety

arXiv AI

1d ago

From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM

arXiv:2608. 15580v1 Announce Type: new Abstract: Reliable endoscopic polyp reporting requires integrating quantitative lesion sizing, standardized Paris classification, and clinically meaningful morphological description within a single record.

By Ruijie Yang, Yan Zhu, Peiyao Fu, Siyuan Li, Te Luo, Zhihua Wang, Quanlin Li, Pinghong Zhou, Xian Yang, Shuo Wang

llms multimodal

arXiv AI

1d ago

Dear Algo: A Precision-First Agentic Intent Layer for Unified Search and Recommendation

arXiv:2608. 15877v1 Announce Type: new Abstract: Search and recommendation serve a shared discovery objective but encode intent differently.

By Rui Wang, Jiazhou Wang, Zheng Wei, Chenglin Lu, Fangcheng Sun, Ivy Sun, Jin Sun, Hui Geng, Lillian Zhang, Chao Yang, Lei Chen, Shahin Sefati, Reem Helou, Joe Zhou, Babak Shakibi, Yiyi Pan, Bi Xue, Hong Yan, Shujian Bu

llms rag agents multimodal

arXiv AI

1d ago

HarmTrace: Anchor-Calibrated Decoupled Optimization for Fine-Grained Target Identification in Harmful Memes

arXiv:2608. 16622v1 Announce Type: cross Abstract: Multimodal harmful meme detection is typically formulated as image--text harmfulness classification.

By Yujia Li, Yiqun Zhang, Zihan Cheng, Yijie Huang, Tenglong Ye, Zihan Wang, Xiaocui Yang, Shi Feng, Yifei Zhang, Daling Wang

llms fine-tuning multimodal

arXiv AI

1d ago

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

arXiv:2608. 16765v1 Announce Type: cross Abstract: Despite recent advances in unified multimodal models for multi-reference image generation, existing benchmarks remain organized around predefined task types (e.

By Haoran Wang, Chaofan Ma, Ran Yi, Lizhuang Ma

diffusion multimodal benchmarks

arXiv AI

1d ago

Neurosymbolic Embodied Agents

arXiv:2608. 16794v1 Announce Type: cross Abstract: Language and vision-language models generate plausible embodied plans but do not guarantee executability, as their outputs can violate environment dynamics or act on incorrectly grounded entities.

By Mohammad Albinhassan, Yuming Feng, Alessandra Russo, Pranava Madhyastha

llms agents robotics multimodal

arXiv AI

1d ago

Euclid-Omni : A Unified Neuro-Symbolic Framework for Plane Geometry

arXiv:2608. 14585v1 Announce Type: new Abstract: Euclidean geometry is a compelling testbed for AI reasoning, as it demands the combination of intuitive diagram understanding, axiomatic deduction, and algebraic computation.

By Zhaoyu Li, Hangrui Bi, Youyuan Zhang, Wenjie Ma, Zenan Li, Zhaolei Zhang, Xujie Si, Kaiyu Yang

llms multimodal benchmarks

arXiv AI

1d ago

EcoVLA: Energy-Efficient Device-Edge Co-Inference for Vision-Language-Action Models under Real-Time Constraints

arXiv:2608. 15502v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have emerged as a promising foundation for Embodied AI, but their high inference cost poses significant challenges for deployment in robotic systems.

By Ao Zhou, Bo Dai, Le Yu, Xingyu Liu, Zeyu Hao, Lingkun Long, Chunming Hu, Jianlei Yang

robotics multimodal

arXiv AI

1d ago

Hypergraph-based Multimodal Retrieval-Augmented Generation with Incremental Refinement

arXiv:2608. 16628v1 Announce Type: new Abstract: Modern Multimodal Retrieval-Augmented Generation (M-RAG) systems are fundamentally limited by the binary connectivity paradigm of traditional simple graphs, which fails to capture the intricate, high-order correlations among heterogeneous entities, such as the N-ary relationships between a visual chart, its scattered textual descriptions, and underlying numerical data.

By Shenao Chen, Yidan Xu, Xiangmin Han, Rundong Xue, Duanpo Wu, Yuhan Gao, Chenggang Yan, Yue Gao

rag multimodal benchmarks

arXiv AI

1d ago

FabriMAE I Trust Myself? Self-Evaluating VLA Action Generation with Markov Attention Entropy

arXiv:2608. 16697v1 Announce Type: new Abstract: Vision-Language-Action models (VLAs) integrate visual perception, language instruction, and action generation into end-to-end policies across heterogeneous architectures.

By Aniri, Chen Yilin, Jinhe Bi, Junfei Guo, Donglai Ran, Xu Bian, Zengjie Jin, Yujun Wang, Yijun Tian, Volker Tresp, Fei Shen, Tat-Seng Chua, Yunpu Ma

multimodal benchmarks

arXiv AI

1d ago

The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning

arXiv:2608. 14558v1 Announce Type: new Abstract: Current multimodal models have demonstrated remarkable proficiency in recognizing static visual and auditory content.

By Garima Arya Yadav, Nilay Yilmaz, Yezhou Yang

llms multimodal benchmarks

arXiv AI

1d ago

Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

arXiv:2608. 14763v1 Announce Type: cross Abstract: Assessment of ventriculomegaly (VM) on fetal brain ultrasound relies primarily on measuring lateral ventricular atrial width on standard planes, which is operator-dependent and may not fully reflect the overall ventricular enlargement.

By Yuhao Huang, Yuanji Zhang, Yuhuan Lu, Dong Ni, P. Ellen Grant, Davood Karimi

llms rag multimodal benchmarks safety

arXiv AI

1d ago

MetaReason: Precise Interleaved Multimodal Reasoning via Editing Meta Information for Solving Geometry Problems

arXiv:2608. 15006v1 Announce Type: cross Abstract: Although visual reasoning is crucial for solving complex geometry tasks, existing vision-language models rely heavily on text-only reasoning.

By Penghao Yin, Haomin Wang, Qihong Tang, Xiaoye Qu, Hongjie Zhang, Xiao-Ping Zhang

llms reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

1d ago

PhaseLoRA: Control-Regime-Conditioned Low-Rank Adaptation for Continuous-Action Vision-Language-Action Policies

arXiv:2608. 15285v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning (PEFT) is a natural way to adapt pretrained vision-language-action (VLA) policies, but most adapter designs apply temporally static updates throughout a control rollout, overlooking the phase-dependent nature of continuous-action manipulation.

By Yufei Guo, Yinan Wu, Haoran Duan, Guiguang Ding, Jungong Han

robotics fine-tuning multimodal

arXiv AI

1d ago

Cross-Domain Industrial Fault Detection by Causal Mechanism Monitoring

arXiv:2608. 14666v1 Announce Type: new Abstract: Unsupervised fault detection in industrial systems is dominated by reconstruction based methods that monitor individual sensor marginal distributions.

By Dhiraj Neupane, Mohamed Reda Bouadjenek, Richard Dazeley, Sunil Aryal

rag multimodal safety

arXiv AI

1d ago

Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning

arXiv:2509. 06461v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have demonstrated remarkable success across diverse visual tasks, yet their performance degrades in complex visual environments.

By Yuyao Ge, Shenghua Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Xuanshan Zhou, Jiayu Yao, Jiafeng Guo, Xueqi Cheng

llms computer-vision multimodal