Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,972 stories · RSS feed

arXiv AI

Jun 30

SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

arXiv:2603. 29139v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have enabled agentic systems to translate natural-language intent into executable scientific visualization (SciVis) tasks.

By Kuangshi Ai, Haichao Miao, Kaiyuan Tang, Nathaniel Gorski, Jianxin Sun, Guoxi Liu, Helgi I. Ingolfsson, David Lenz, Hanqi Guo, Hongfeng Yu, Teja Leburu, Michael Molash, Bei Wang, Tom Peterka, Chaoli Wang, Shusen Liu

llms agents multimodal benchmarks

arXiv AI

Jun 30

SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models' performance

arXiv:2606. 30113v1 Announce Type: cross Abstract: Discrete action tokenization provides a compact interface for autoregressive VLA policies, but accurately recovering continuous robot actions from discrete codes remains challenging.

By Tengyue Jiang, Chunpu Xu, Jiayue Kang, Yao Mu

llms nlp robotics multimodal

arXiv AI

Jun 30

TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech

arXiv:2606. 30543v1 Announce Type: cross Abstract: With the proliferation of speech AI agents, understanding emotional entrainment in conversational interaction has become increasingly important.

By Sathvik Manikantan Napa Ugandhar, Hao Zhang, Alison Gunzler, Yuzhe Wang, Thomas Thebaud, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Vel\'azquez

rag agents fine-tuning multimodal

arXiv AI

Jun 30

Learning to Adaptively Allocate Gaussians for Arbitrary-Scale Image Super-Resolution

arXiv:2606. 29400v1 Announce Type: cross Abstract: In computer graphics, visual content is continuously warped, zoomed and resampled.

By Giulio Federico, Giuseppe Amato, Claudio Gennaro, Fabio Carrara, Marco Di Benedetto

diffusion efficiency multimodal benchmarks

arXiv AI

Jun 30

When Stopping Fails: Rethinking Minimal Risk Conditions through Human-Interactive Autonomous Driving for Safe Transportation Systems

arXiv:2606. 29115v1 Announce Type: cross Abstract: Autonomous vehicles (AVs) are increasingly deployed in urban environments, yet their safety frameworks remain primarily designed around collision avoidance and minimal risk condition (MRC) behaviors such as slowing or stopping when uncertainty arises.

By Yash Tandon, Giovanni Tapia Lopez, Marcus Blennemann, Mohan Trivedi, Ross Greer

agents robotics multimodal safety

arXiv AI

Jun 30

Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition

arXiv:2605. 13087v2 Announce Type: replace-cross Abstract: Fine-tuning multilingual ASR models like Whisper for low-resource languages often improves read speech but degrades spontaneous audio performance.

By Kush Juvekar, Kavya Manohar, Aditya Srinivas Menon, Arghya Bhattacharya, Kumarmanas Nethil

fine-tuning multimodal benchmarks

arXiv AI

Jun 30

Attribution Graphs and Causal Probing for Mechanistic Discovery and Bias Repair in Multimodal Generative Learning

arXiv:2510. 12957v4 Announce Type: replace-cross Abstract: We treat the internals of generative models as mechanistic objects rather than black boxes.

By Noor Islam S. Mohammad, Ulu\u{g} Bayaz{\i}t

fine-tuning multimodal safety

arXiv AI

Jun 30

Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

arXiv:2606. 29023v1 Announce Type: cross Abstract: Spatio-temporal grounding in long videos requires precise temporal localization and robust object tracking conditioned on natural-language queries.

By Tianshu Zhang, Yan Wang, Ji Qi, Lijie Wen

llms reinforcement-learning multimodal

arXiv AI

Jun 30

StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

arXiv:2507. 07445v3 Announce Type: replace Abstract: Autonomous agents navigating human society must master both production activities and social interactions, yet existing benchmarks rarely evaluate these skills simultaneously.

By Weihao Tan, Changjiu Jiang, Yu Duan, Mingcong Lei, Jiageng Li, Yitian Hong, Xinrun Wang, Bo An

llms agents robotics multimodal benchmarks

arXiv AI

Jun 30

Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

arXiv:2606. 29984v1 Announce Type: new Abstract: Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs).

By Peng, Lee, Yin Zhang, Yanglin Zhang, Haonan Wu, Zishan Liu, Ruoxi Zang, Xin Zhu, Jiayin Zheng, Jian Yao, Zefeng Ji, Fei Ma

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jun 30

PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation

arXiv:2606. 28344v1 Announce Type: cross Abstract: Augmenting large language models (LLMs) with retrieved web text has become a dominant paradigm, yet the web is not natively textual: existing systems depend on complex parsing pipelines that linearize HTML and discard layout, visual structure, and formatting.

By Yichuan Wang, Zhifei Li, Zirui Wang, Paul Teiletche, Lesheng Jin, Matei Zaharia, Joseph E. Gonzalez, Sewon Min

llms rag agents multimodal benchmarks

arXiv AI

Jun 30

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

arXiv:2606. 28385v1 Announce Type: cross Abstract: Recent advances in robot world models enable synthetic video generation for embodied prediction and planning.

By Minh-Loi Nguyen, Nghiem Tuong Diep, Hung Khang Nguyen, Minh Le, Doanh Le Thien, Hoang H. Tran, Dung D. Le, Vu N. Duong, Daniel Sonntag, An Thai Le, Duy Minh Ho Nguyen, Vien Anh Ngo, Tran Van Nhiem

llms agents diffusion robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 30

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

arXiv:2606. 28396v1 Announce Type: cross Abstract: Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories.

By Emily Bejerano, Federico Tondolo, Devang Gupta, Aaron Mano Cherian, Taeyoo Kim, Ayaan Qayyum, Xiaofan Yu, Xiaofan Jiang

llms robotics multimodal

arXiv Machine Learning

Jun 30

From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderation

arXiv:2606. 30059v1 Announce Type: new Abstract: Industry-scale video and live-streaming moderation imposes requirements that are difficult to satisfy with generic pretrained public models or external APIs, including adaptation to platform-specific data distributions, policy-specific objectives, and product-level safety constraints.

By Shuchang Ye, Jinqiang Yu, Zhujun Xiao, Yajing Kong, Yist Y. Lin, Yang Ma, Jiaxi Liu, Xiaolei Xu, Zheng Yu

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 30

A Diagnostic Framework and Multi-Evaluator Audit of Evaluator-Driven Preference Dynamics in Self-Adapting LLM Agents

arXiv:2606. 29719v1 Announce Type: new Abstract: Measurements of proprietary LLM evaluators can become invalid within weeks -- we document one case and provide the diagnostic framework to detect it.

By Liu Zewen

llms agents multimodal benchmarks

arXiv AI

Jun 30

Unlocking the Visual Record of Materials Science: A Large-Scale Multimodal Dataset from Scientific Literature

arXiv:2606. 29667v1 Announce Type: cross Abstract: The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale.

By Subham Ghosh, Shubham Tiwari, Mohammad Ibrahim, Abhishek Tewari

llms fine-tuning multimodal safety

arXiv AI

Jun 30

Demonstration-Free Robotic Control via LLM Agents

arXiv:2601. 20334v2 Announce Type: replace-cross Abstract: Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift.

By Brian Y. Tsui, Alan Y. Fang, Tiffany J. Hwu

llms agents robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 30

EfficientUICoder: A Bidirectional Token Compression Framework for Efficient MLLM-Based UI Code Generation

arXiv:2509. 12159v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models have demonstrated exceptional performance in UI2Code tasks, significantly enhancing website development efficiency.

By Jingyu Xiao, Zhongyi Zhang, Yuxuan Wan, Yintong Huo, Yang Liu, Michael R. Lyu

llms multimodal

arXiv AI

Jun 30

Primary ICD Category Prediction using LLM-based Probing

arXiv:2606. 28798v1 Announce Type: new Abstract: Objective: ICD codes are central to reimbursement, research, and population health surveillance, yet automated coding systems often struggle to integrate diagnostic signals from both clinical narratives and structured electronic health record (EHR) variables.

By Chengyuan Liu, Xinyue Zhang, Yao Li, Guanting Chen

llms rag multimodal

arXiv AI

Jun 30

LLM-Guided Planning for Multi-hop Reasoning over Multimodal Nuclear Regulatory Documents

arXiv:2606. 29399v1 Announce Type: new Abstract: Reviewing nuclear regulatory documents requires multi-hop reasoning across tens of thousands of pages, where judgments depend on evidence assembled across multiple chapters.

By Mingyu Jeon, Bokyeong Kim, Suwan Cho, Jae Young Suh, Yonggyun Yu

llms agents multimodal benchmarks safety

Get the daily briefing