Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

3,800 stories · RSS feed

arXiv AI

1d ago

Dear Algo: A Precision-First Agentic Intent Layer for Unified Search and Recommendation

arXiv:2608. 15877v1 Announce Type: new Abstract: Search and recommendation serve a shared discovery objective but encode intent differently.

By Rui Wang, Jiazhou Wang, Zheng Wei, Chenglin Lu, Fangcheng Sun, Ivy Sun, Jin Sun, Hui Geng, Lillian Zhang, Chao Yang, Lei Chen, Shahin Sefati, Reem Helou, Joe Zhou, Babak Shakibi, Yiyi Pan, Bi Xue, Hong Yan, Shujian Bu

llms rag agents multimodal

arXiv AI

1d ago

Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

arXiv:2608. 14763v1 Announce Type: cross Abstract: Assessment of ventriculomegaly (VM) on fetal brain ultrasound relies primarily on measuring lateral ventricular atrial width on standard planes, which is operator-dependent and may not fully reflect the overall ventricular enlargement.

By Yuhao Huang, Yuanji Zhang, Yuhuan Lu, Dong Ni, P. Ellen Grant, Davood Karimi

llms rag multimodal benchmarks safety

arXiv AI

1d ago

Physiological World Models for Human State Transitions

arXiv:2608. 15309v1 Announce Type: new Abstract: Continuous multimodal sensing now allows human physiology to be observed throughout daily life rather than only during occasional clinical visits.

By Chongyang Zhang, Rendong Wang, Hao Zheng, Hanwen Zhang, Yang Liu, Xiaolong Wei, Bin Chong

multimodal benchmarks safety data-engineering

arXiv AI

1d ago

Toward AI-Friendly Cartography: Understanding How Color Design Influences Foundation Model Spatial Reasoning on Sequential Choropleth Maps

arXiv:2608. 15736v1 Announce Type: new Abstract: Foundation models (FMs) increasingly support multimodal and geospatial reasoning, yet it remains unclear whether cartographic principles designed for human perception are equally effective for machines.

By Yonghe Sun, Zhenjia Liu, Hua Liao, Wenjia Xu, Nai Yang, Weihua Dong, Zhiwei Wei

fine-tuning multimodal benchmarks

arXiv AI

1d ago

Hypergraph-based Multimodal Retrieval-Augmented Generation with Incremental Refinement

arXiv:2608. 16628v1 Announce Type: new Abstract: Modern Multimodal Retrieval-Augmented Generation (M-RAG) systems are fundamentally limited by the binary connectivity paradigm of traditional simple graphs, which fails to capture the intricate, high-order correlations among heterogeneous entities, such as the N-ary relationships between a visual chart, its scattered textual descriptions, and underlying numerical data.

By Shenao Chen, Yidan Xu, Xiangmin Han, Rundong Xue, Duanpo Wu, Yuhan Gao, Chenggang Yan, Yue Gao

rag multimodal benchmarks

arXiv AI

1d ago

Privacy-Preserving Dataset Curation for Kuala Lumpur Urban Traffic: Grounded Vision-Language Detection with Spatial Vehicle-Context Filtering

arXiv:2608. 14724v1 Announce Type: cross Abstract: The rapid advancement of intelligent transportation systems and autonomous driving relies heavily on multi-modal urban traffic datasets.

By Mohammed Abdul Al Arafat Tanzin, Rudzidatul Akmam Dziyauddin

llms agents robotics multimodal safety

arXiv AI

1d ago

Evaluating Multimodal LLMs across Text and Audio Modalities for Accessible Disaster Assistance

arXiv:2608. 14651v1 Announce Type: new Abstract: Effective disaster risk communication is a foundational humanitarian challenge, yet current emergency infrastructure fails to meet the needs of individuals with access and functional needs, including hard-of-hearing individuals, pregnant women, mothers with toddlers, and elderly individuals with dementia.

By Anuridhi Gupta, Samara Mansoor, Hemant Purohit

llms multimodal benchmarks

arXiv AI

1d ago

VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

arXiv:2608. 15265v1 Announce Type: new Abstract: Constructing an interactive 3D open world from a user query is important.

By Yansong Ning, Jingwen Ye, Zhongkai Wu, Yang Sun, Yiqin Zhu, Xingyi Li, Weidong Zhang, Hao Liu

llms agents reinforcement-learning multimodal benchmarks

arXiv AI

1d ago

From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM

arXiv:2608. 15580v1 Announce Type: new Abstract: Reliable endoscopic polyp reporting requires integrating quantitative lesion sizing, standardized Paris classification, and clinically meaningful morphological description within a single record.

By Ruijie Yang, Yan Zhu, Peiyao Fu, Siyuan Li, Te Luo, Zhihua Wang, Quanlin Li, Pinghong Zhou, Xian Yang, Shuo Wang

llms multimodal

arXiv AI

1d ago

TRACE: Trajectory Aware Reasoning for Multi-Turn Adversarial Conversation Evaluation

arXiv:2608. 15594v1 Announce Type: new Abstract: Multi-turn jailbreak attacks have emerged as a critical safety threat to LLMs, as harmful objectives are decomposed across a sequence of apparently benign turns to bypass guardrails.

By Md Messal Monem Miah, Adrita Anika, Zhiyuan Yu, Ruihong Huang

llms robotics fine-tuning multimodal benchmarks safety

arXiv AI

1d ago

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

arXiv:2608. 15930v1 Announce Type: new Abstract: Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution.

By Zihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke, Mengcheng Lan, Weixian Lei, Hanxuan Li, Honglin Li, Xiyun Li, Zaitang Li, Leowei Liang, Xin Luo, Haozhe Ma, Jiayi Mao, Zhoujie Pan, Can Qin, Tianyuan Qu, Weiqi Wang, Wenkai Wang, Yonglin Wang, Yuxin Wang, Chenxu Wu, Yingchen Yu, Chenyu Zhang, Yuhao Zheng

agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

1d ago

MUPA$^{2}$E: Multimodal Unified Perception with Asymmetric Attention for Emotion Assessment

arXiv:2608. 15999v1 Announce Type: new Abstract: Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion.

By Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz, Randy Gomez

multimodal

arXiv AI

1d ago

FabriMAE I Trust Myself? Self-Evaluating VLA Action Generation with Markov Attention Entropy

arXiv:2608. 16697v1 Announce Type: new Abstract: Vision-Language-Action models (VLAs) integrate visual perception, language instruction, and action generation into end-to-end policies across heterogeneous architectures.

By Aniri, Chen Yilin, Jinhe Bi, Junfei Guo, Donglai Ran, Xu Bian, Zengjie Jin, Yujun Wang, Yijun Tian, Volker Tresp, Fei Shen, Tat-Seng Chua, Yunpu Ma

multimodal benchmarks

arXiv AI

1d ago

Path2ST: Hierarchical Cell-Tissue Grounded Cross-Modal Translation for Spatial Transcriptomics

arXiv:2608. 14710v1 Announce Type: cross Abstract: Predicting spatial gene expression from hematoxylin and eosin (H\&E)-stained images offers a cost-effective alternative to spatial transcriptomics (ST).

By Ruochen Liu, Wei Lou

multimodal benchmarks

arXiv AI

1d ago

The Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual Reasoning

arXiv:2608. 14558v1 Announce Type: new Abstract: Current multimodal models have demonstrated remarkable proficiency in recognizing static visual and auditory content.

By Garima Arya Yadav, Nilay Yilmaz, Yezhou Yang

llms multimodal benchmarks

arXiv AI

1d ago

Euclid-Omni : A Unified Neuro-Symbolic Framework for Plane Geometry

arXiv:2608. 14585v1 Announce Type: new Abstract: Euclidean geometry is a compelling testbed for AI reasoning, as it demands the combination of intuitive diagram understanding, axiomatic deduction, and algebraic computation.

By Zhaoyu Li, Hangrui Bi, Youyuan Zhang, Wenjie Ma, Zenan Li, Zhaolei Zhang, Xujie Si, Kaiyu Yang

llms multimodal benchmarks

arXiv AI

1d ago

Cross-Domain Industrial Fault Detection by Causal Mechanism Monitoring

arXiv:2608. 14666v1 Announce Type: new Abstract: Unsupervised fault detection in industrial systems is dominated by reconstruction based methods that monitor individual sensor marginal distributions.

By Dhiraj Neupane, Mohamed Reda Bouadjenek, Richard Dazeley, Sunil Aryal

rag multimodal safety

arXiv AI

1d ago

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

arXiv:2608. 14841v1 Announce Type: new Abstract: Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines.

By Guanchen Wu, Jiayuan Ding, Subhabrata Mukherjee, Carl Yang

llms nlp multimodal

arXiv AI

1d ago

EcoVLA: Energy-Efficient Device-Edge Co-Inference for Vision-Language-Action Models under Real-Time Constraints

arXiv:2608. 15502v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have emerged as a promising foundation for Embodied AI, but their high inference cost poses significant challenges for deployment in robotic systems.

By Ao Zhou, Bo Dai, Le Yu, Xingyu Liu, Zeyu Hao, Lingkun Long, Chunming Hu, Jianlei Yang

robotics multimodal

arXiv AI

1d ago

Workspace Topology as an Attack Vector in Agentic Coding Assistants

arXiv:2608. 14876v1 Announce Type: cross Abstract: Agentic coding assistants are finding widespread use, not just in new code development but in quickly ingesting and leveraging third-party code.

By Alexandre G. R. Day, Pradeep Yadlapalli, Sriram Venkatapathy, Thomas Paniagua, Nick Raines, Sahil Wadhwa, Himanshu Kumar, Andy Luo, Sudeep Panyam, Rikhiya Ghosh, Pranab Mohanty, Giri Iyengar

agents multimodal safety