Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,022 stories · RSS feed

arXiv AI

Jul 22

Appearance Pointers -- Multimodal Region Control of Diffusion Transformers

arXiv:2607. 19344v1 Announce Type: cross Abstract: Controllable image generation remains challenging for creative professionals, who often require precise regional control over materials, object identities, and spatial arrangements that cannot be reliably achieved through text prompting alone.

By Rahul Sajnani, Yulia Gryaditskaya, Radom\'ir M\v{e}ch, Srinath Sridhar, Matheus Gadelha

llms diffusion multimodal

arXiv AI

Jul 22

GUIDED Network-Agnostic Feature Initialization for Spatial Transferability in GNN-based Models

arXiv:2607. 19270v1 Announce Type: cross Abstract: The Traffic Assignment Problem is a fundamental but computationally expensive component of transportation planning.

By Alessandro Scalese, Santhanakrishnan Narayanan, Constantinos Antoniou

rag fine-tuning multimodal benchmarks

arXiv AI

Jul 22

FormGym: Doing Paperwork with Agents

arXiv:2506. 14079v4 Announce Type: replace Abstract: Completing paperwork is a challenging and time-consuming problem.

By Matthew Toles, Rattandeep Singh, Isaac Song, Zhou Yu

llms agents computer-vision multimodal benchmarks

arXiv AI

Jul 22

MAGE: Human-Like Macro Placement via Agentic Multimodal Reasoning

arXiv:2607. 18536v1 Announce Type: new Abstract: Macro placement still requires substantial manual refinement in industrial physical design flows.

By Andrew B. Kahng, Sayak Kundu, Bodhisatta Pramanik

agents multimodal safety

arXiv AI

Jul 22

What the Waveform Knows: Transparent-first Speech and Audio Intelligence with Caption Studio

arXiv:2607. 18704v1 Announce Type: cross Abstract: Caption Studio is a transparency-first speech and audio intelligence platform that transforms spoken audio and video into structured, searchable content through automated transcription, speaker diarization, speech analytics, signal-level audio analysis, and subtitle generation.

By Cheng Siong Chin, Jianhua Zhang, Mohan Venkateshkumar

multimodal benchmarks safety

arXiv AI

Jul 22

ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU

arXiv:2607. 19191v1 Announce Type: cross Abstract: We present ABot-World-0, an action-conditioned video world model for real-time, long-horizon closed-loop interaction, supported by a multi-source data infrastructure spanning AAA games, simulation engines, and internet videos to learn controllable world dynamics.

By Fan Jiang, Zhaoxu Sun, Mengchao Wang, Ziyu Zhu, Chiyu Wang, Yunpeng Zhang, Wenlin Liu, Yun Wang, Xue Zheng, Rui Sun, Junfeng Ni, Hongyu Pan, Zhongxu Sun, Fei Yu, Zengye Ge, Mengmeng Du, Nianfei Fan, Mingchao Sun, Yu Liu, Yongchang, Yanqing Zhu, Jiahang Wang, Ning Ying, Yuze Xuan, Di Yang, Zhicheng Liu, Zhe Gao, Tingbing Xu, Jiacheng Sui, Wenjin Yang, Junnan Lai, Shufeng Liu, Yuan Liu, Zheng Zhou, Yingliang Peng, Dawei Cao, Kaifeng Sheng, Yuxiang Cai, Fei Lu, Mu Xu, Ning Guo

agents efficiency multimodal

arXiv AI

Jul 22

Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models

arXiv:2607. 18695v1 Announce Type: cross Abstract: A popular route to interpretable zero-shot classification asks a large language model (LLM) to describe each class name and prompts CLIP with the resulting descriptors.

By Gautam Rajendrakumar Gare, Jia Shi, Zhiqiu Lin, Deepak Pathak, John Galeotti, Deva Ramanan

llms rag multimodal

arXiv AI

Jul 22

Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety

arXiv:2607. 18828v1 Announce Type: new Abstract: Readiness stress-testing of medical AI has focused on closed-ended and multimodal benchmarks.

By Koyar Afrasyab

llms multimodal benchmarks safety

arXiv AI

Jul 22

OPD-IAD: From Language Judgment to Industrial Anomaly Detection via On-Policy Self-Distillation

arXiv:2607. 18850v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) have recently shown strong potential for industrial anomaly detection (IAD) by providing image-level anomaly judgments and interpretable defect reasoning.

By Shuimu Chen, Jing Jin, Nan Su, Hongbo Xu, Zebang Cheng, Wenming Yang, Fei Ma, Guijin Wang

llms efficiency multimodal

arXiv AI

Jul 22

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

arXiv:2607. 19190v1 Announce Type: cross Abstract: Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation.

By Guanxiong Chen, Qianjun Xia, Jiawei Peng, Heng Zhang, Bole Ma, Justin Qian, Ziyi Jiao, Bingyang Zhou, Luoxin Ye, Kaifeng Zhang, Kunyi Wang, Weijia Zeng, Yunuo Chen, Pengzhi Yang, Ziqiu Zeng, Huamin Wang, Chao Liu, Alan Yuille, Fan Shi, Changxi Zheng, Yunzhu Li, Chenfanfu Jiang, Peter Yichen Chen

agents robotics multimodal safety

arXiv AI

Jul 22

Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains

arXiv:2607. 18438v1 Announce Type: cross Abstract: Introducing Relay-Bench, an unsaturated, holistic, text-only benchmark that measures LLMs' ability to complete an assortment of tasks from distinct domains in a single prompt.

By Liam Swayne

llms multimodal benchmarks

arXiv AI

Jul 22

Quality Action Assurance: Multimodal Verification of Examiner Claims in VR OSCEs

arXiv:2607. 19063v1 Announce Type: new Abstract: Objective Structured Clinical Examinations (OSCEs) are the gold standard for assessing clinical competence, yet scoring remains vulnerable to examiner subjectivity, fatigue, and cognitive bias.

By Harry Rogers, Sally Shiels, Ashley Tomlinson, James Thomas, James Aylward, Nathan Gauge, Helen Higham, Alison Noble

llms multimodal safety

arXiv Machine Learning

Jul 22

Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

arXiv:2607. 18615v1 Announce Type: cross Abstract: Machine unlearning for vision-language models (VLMs) remains underexplored.

By Zijie Liu, Jinhao Duan, Gaowen Liu, Sijia Liu, Tianlong Chen

llms multimodal

arXiv AI

Jul 22

When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Models

arXiv:2603. 03989v2 Announce Type: replace-cross Abstract: When visual evidence is ambiguous, vision models must decide how to interpret face-like patterns.

By Qianpu Chen, Derya Soydaner, Rob Saunders

llms computer-vision multimodal safety

arXiv Machine Learning

Jul 22

One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models

arXiv:2607. 19128v1 Announce Type: new Abstract: Vision-language models (VLMs) provide a unified representation space for textual and visual information, yet their potential as general-purpose backbones for graph-structured data remains largely unexplored.

By Jiayi Yang, Yifang Chen, Yuanfu Sun, Jiajin Liu, Qiaoyu Tan

llms rag multimodal benchmarks

arXiv AI

Jul 22

PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image

arXiv:2607. 19261v1 Announce Type: cross Abstract: Whole-slide image (WSI) diagnosis requires identifying diagnostically relevant regions, examining them across magnifications, and integrating multi-scale evidence.

By Dankai Liao, Tianyi Zhang, Yufeng Wu, Xinyue Zhang, Qiaochu Xue, Zeyu Liu, Dachun Zhao, Linghan Cai, Yueming Jin

llms agents diffusion multimodal benchmarks

arXiv AI

Jul 22

AnchorRefine: Synergy-Manipulation Based on Trajectory Anchor and Residual Refinement for Vision-Language-Action Models

arXiv:2604. 17787v2 Announce Type: replace-cross Abstract: Precision-critical manipulation requires both global trajectory organization and local execution correction, yet most vision-language-action (VLA) policies generate actions within a single unified space.

By Tingzheng Jia, Kan Guo, Lanping Qian, Yongli Hu, Daxin Tian, Guixian Qu, Chunmian Lin, Baocai Yin, Jiapu Wang

diffusion robotics multimodal

arXiv AI

Jul 22

Why Do Vision Language Models Struggle To Recognize Human Emotions?

arXiv:2604. 15280v2 Announce Type: replace-cross Abstract: Understanding emotions is a fundamental ability for intelligent systems to be able to interact with humans.

By Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara, Steven McDonagh

llms multimodal safety

arXiv Machine Learning

Jul 22

EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation

arXiv:2607. 18336v1 Announce Type: cross Abstract: Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance.

By Zilong Huang, Kong Aik Lee, Junjie Li, Zhe Li, Man-Wai Mak

multimodal benchmarks

arXiv AI

Jul 22

Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing

arXiv:2607. 19064v1 Announce Type: cross Abstract: Large-scale visual generators are increasingly capable but costly to train, fine-tune, and deploy.

By Xinjie Zhang, Peng Zhang, Shicheng Zheng, Jinghao Guo, Zhaoyang Jia, Yifei Shen, Xun Guo, Yuxuan Luo, Jiahao Li, Wenxuan Xie, Fanyi Pu, Xiaoyi Zhang, Kaichen Zhang, Zongyu Guo, Tianci Bi, Dongnan Gui, Zhening Liu, Zimo Wen, Zihan Zheng, Senqiao Yang, Xiao Li, Jinglu Wang, Bin Li, Yan Lu

llms diffusion nlp reinforcement-learning fine-tuning efficiency multimodal benchmarks safety

Get the daily briefing