Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,498 stories · RSS feed

arXiv AI

Jun 30

A Multi-Dataset Benchmark for Evaluating LLM Agents in Microservice Failure Diagnosis

arXiv:2606. 29193v1 Announce Type: cross Abstract: LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data.

By Yuanhong Cai, Xiaohui Nie, Kanglin Yin, Changhua Pei, Yongqian Sun, Shenglin Zhang, Haibin Liu, Guiyang Liu, Xidao Wen, Fang Situ, Dan Pei

llms agents multimodal benchmarks

arXiv AI

Jun 30

Digitizing Coaching Intelligence: An Agentic Framework for Holistic Athlete Profiling using VLM and RAG

arXiv:2606. 28570v1 Announce Type: cross Abstract: Athlete assessment is a critical process for tracking physical progress and identifying elite talent.

By Deep Ghosal, Ishani Sen, Wazib Ansar, Amlan Chakrabarti

llms rag agents computer-vision multimodal safety data-engineering

arXiv Machine Learning

Jun 30

MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

arXiv:2606. 29788v1 Announce Type: new Abstract: When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success.

By Kuan Wang, Chao Zhang

agents multimodal benchmarks

arXiv AI

Jun 30

Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction

arXiv:2606. 29949v1 Announce Type: cross Abstract: H&E-stained whole-slide images offer cohort-scale availability and rich spatial context but lack molecular specificity, whereas bulk RNA-seq provides transcriptome-wide resolution at high cost with limited archival availability.

By Dominik Winter, Dominik Vonficht, Lo\"ic Le Bescond, Christian Gebbe, Marco Rosati, Richard J. Chen, Markus Schick, Ross Stewart, Nicolas Brieu

llms fine-tuning multimodal safety

arXiv AI

Jun 30

Dual-Flow Reinforcement Learning with State-Aware Exploration

arXiv:2606. 29820v1 Announce Type: cross Abstract: In complex continuous-control reinforcement learning tasks, multimodal optimal actions often coincide with uncertain, multimodal return distributions, making reliable value estimation and multimodal exploration challenging.

By Qijun Li, Zheng Fu, Qi Song, Yifei He, Weitao Zhou, Kun Jiang, Diange Yang

diffusion reinforcement-learning multimodal benchmarks

arXiv AI

Jun 30

PromptGNN-sim: Deep Fusion and Alignment of GNN and LLMs for Text-Attributed Graph Learning

arXiv:2606. 30291v1 Announce Type: new Abstract: Text-Attributed Graphs (TAGs) combine textual semantics with graph structure and are central to many graph learning tasks.

By Zhifei Hu, Alexandra I. Cristea

llms multimodal safety

arXiv AI

Jun 30

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

arXiv:2603. 22282v2 Announce Type: replace-cross Abstract: We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture.

By Ziyi Wang, Xinshun Wang, Shuang Chen, Yang Cong, Mengyuan Liu

llms nlp efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Contrastive vision-language learning with paraphrasing and negation

arXiv:2511. 16527v2 Announce Type: replace-cross Abstract: Contrastive vision-language models continue to be the dominant approach for image-text retrieval.

By Kwun Ho Ngan, Saman Sadeghi Afgeh, Joe Townsend, Artur d'Avila Garcez

llms rag diffusion computer-vision multimodal

arXiv AI

Jun 30

The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis

arXiv:2606. 29581v1 Announce Type: cross Abstract: Modern LLM deployments routinely compress models and raise sampling temperature to reduce cost, latency, or repetition, yet safety evaluations usually treat these choices as fixed implementation details.

By Hari Prasad, Ritam Pal

llms efficiency multimodal safety

arXiv AI

Jun 30

X-Mind: Efficient Visual Chain-of-Thought via Predictive World Model for End-to-End Driving

arXiv:2606. 28758v1 Announce Type: cross Abstract: Predicting future states is essential for autonomous agents, yet current Vision-Language-Action (VLA) models fundamentally lack this capability, relying instead on reactive perception-action mapping.

By Bohao Zhao, Chengrui Wei, Guangfeng Jiang, Ruixin Liu, Xuejie Lv, Liu Liang, Sutao Deng, Xiuyang Fan, Pengkun Zheng, Jinyun Zhou, Rui Guo, Hanpeng Liu, Yutong Zheng, Yi Guo, Xinlong Zheng, Qingyu Luo, Zhuangzhuang Ding, Yu Zhang, Hang Zhang, Xianming Liu

agents diffusion multimodal

arXiv AI

Jun 30

Learning to Adaptively Allocate Gaussians for Arbitrary-Scale Image Super-Resolution

arXiv:2606. 29400v1 Announce Type: cross Abstract: In computer graphics, visual content is continuously warped, zoomed and resampled.

By Giulio Federico, Giuseppe Amato, Claudio Gennaro, Fabio Carrara, Marco Di Benedetto

diffusion efficiency multimodal benchmarks

arXiv Machine Learning

Jun 30

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

arXiv:2606. 28406v1 Announce Type: new Abstract: Text-to-image and multimodal generative models are increasingly used to produce scientific figures such as mechanism diagrams, experimental-design schematics, conceptual frameworks, and graphical abstracts.

By Davie Chen

diffusion computer-vision multimodal benchmarks

arXiv AI

Jun 30

Unlocking the Visual Record of Materials Science: A Large-Scale Multimodal Dataset from Scientific Literature

arXiv:2606. 29667v1 Announce Type: cross Abstract: The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale.

By Subham Ghosh, Shubham Tiwari, Mohammad Ibrahim, Abhishek Tewari

llms fine-tuning multimodal safety

arXiv AI

Jun 30

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

arXiv:2606. 29879v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning.

By Chen Yang, Yuhao Wei, Ze Xu, Ziheng Zou, Shuang Liang, Delin Ouyang, Lingfeng Qi, Jie Li, Guofa Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 30

FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

arXiv:2606. 29431v1 Announce Type: new Abstract: Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image.

By Yichen Guo, Kai Tang, Fenglai Lin, Yiding Sun, Dongshuo Zhang, Wenya Wang, Lin William Cong, Shanghang Zhang

llms multimodal benchmarks safety

arXiv AI

Jun 30

LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion

arXiv:2606. 29900v1 Announce Type: cross Abstract: Personality recognition in asynchronous video interviews (AVIs) has become increasingly important due to their widespread adoption in modern recruitment.

By Tianyi Zhang, Wei Shan, Yuan Zong, Tianhua Qi, Wenming Zheng

llms rag multimodal benchmarks safety

arXiv AI

Jun 30

Automating the Design of Embodied AgentArchitectures

arXiv:2606. 30111v1 Announce Type: cross Abstract: Embodied agents are typically built as hand-designed compositions of perception, memory, planning, and action modules.

By Jian Zhou, Sihao Lin, Jin Li, Shuai Fu, Gengze Zhou, Qi Wu

agents nlp robotics efficiency multimodal

arXiv AI

Jun 30

SwarmX: Agentic Scheduling for Low-Latency Agentic Systems

arXiv:2606. 21401v2 Announce Type: replace-cross Abstract: Agentic AI applications compose multiple model calls and tool executions, creating new scheduling challenges for GPU-CPU clusters.

By Yeqi Huang, Yanwei Ye, Guomin Chen, Wenhao Su, Bin Gong, Jialian Li, Zhan Lu, Yangshen Deng, Xuan Sun, Le Xu, Luo Mai

agents efficiency multimodal benchmarks

arXiv AI

Jun 30

MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

arXiv:2606. 30026v1 Announce Type: cross Abstract: Audiovisual arts encompass diverse creative disciplines, including cinema, visual arts, stage performance, and game design, where artistic meaning arises from deliberate combinations of visual, auditory, and narrative elements (e.

By Yuxuan Fan, Gyusik Seo, Jing Hao, Jaemin Cho, Mohit Bansal, Jaehong Yoon

llms multimodal benchmarks safety

arXiv AI

Jun 30

Forewarned is Forearmed: When Non-Sequential Embedding Turns Into an Anomaly Detector

arXiv:2606. 30196v1 Announce Type: cross Abstract: This paper offers an in-depth analysis of non-sequential multimodal sentence-level embeddings, with a particular focus on the SONAR model.

By Elys Allesiardo, Antoine Caubri\`ere, Valentin Vielzeuf

rag multimodal

Get the daily briefing