Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,470 stories · RSS feed

arXiv AI

Jun 2

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

arXiv:2606. 02578v1 Announce Type: cross Abstract: Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers.

By Seojeong Park, Jiho Choi, Junyong Kang, Seonho Lee, Jaeyo Shin, Hyunjung Shim

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 2

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

arXiv:2603. 24511v2 Announce Type: replace-cross Abstract: We show that AI agents are capable of discovering novel algorithms for adversarial attacks against LLMs, advancing the state of the art on white-box jailbreaking and prompt injection evaluations.

By Alexander Panfilov, Peter Romov, Igor Shilov, Yves-Alexandre de Montjoye, Jonas Geiping, Maksym Andriushchenko

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

ProductWebGen: Benchmarking Multimodal Product Webpage Generation

arXiv:2606. 01022v1 Announce Type: cross Abstract: Crafting a product display webpage from a source product image, along with layout and visual content instructions, holds significant practical value for domains such as marketing, advertising, and E-commerce.

By Zhihong Liu, Siqi Kou, Zheng Li, Ye Ma, Quan Chen, Peng Jiang, Kai Yu, Zhijie Deng

llms diffusion fine-tuning multimodal benchmarks

arXiv AI

Jun 2

EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams

arXiv:2603. 27223v2 Announce Type: replace-cross Abstract: We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union.

By Jaeseong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee

llms multimodal benchmarks

arXiv AI

Jun 2

Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion

arXiv:2602. 23694v3 Announce Type: replace-cross Abstract: Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential.

By Seungyeol Baek, Jaspreet Singh, Lala Shakti Swarup Ray, Hymalai Bello, Paul Lukowicz, Sungho Suh

robotics multimodal benchmarks safety

arXiv AI

Jun 2

CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects

arXiv:2510. 14904v4 Announce Type: replace-cross Abstract: Dense Video Object Captioning (DVOC) is the task of jointly detecting, tracking, and captioning object trajectories in a video, requiring the ability to understand spatio-temporal details and describe them in natural language.

By Gabriel Fiastre, Antoine Yang, Cordelia Schmid

computer-vision multimodal benchmarks

arXiv AI

Jun 2

From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model

arXiv:2512. 05277v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances.

By Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari

llms agents robotics multimodal benchmarks safety

arXiv AI

Jun 2

Video Reasoning without Training

arXiv:2510. 17045v2 Announce Type: replace-cross Abstract: Video reasoning using Large Multimodal Models (LMMs) relies on costly reinforcement learning (RL) and verbose chain-of-thought, resulting in substantial computational overhead during both training and inference.

By Deepak Sridhar, Kartikeya Bhardwaj, Jeya Pradha Jeyaraj, Nuno Vasconcelos, Ankita Nayak, Harris Teague

reinforcement-learning fine-tuning multimodal

arXiv AI

Jun 2

Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation

arXiv:2601. 00664v2 Announce Type: replace-cross Abstract: Talking head generation creates lifelike avatars from static portraits for virtual communication and content creation.

By Taekyung Ki, Sangwon Jang, Jaehyeong Jo, Jaehong Yoon, Sung Ju Hwang

diffusion multimodal

arXiv AI

Jun 2

MASER: Modality-Adaptive Specialist Routing for Embodied 3D Spatial Intelligence

arXiv:2606. 02463v1 Announce Type: cross Abstract: In 3D environments, Embodied Agents answer spatially relevant questions through reasoning from a mixture of modalities including natural language, RGB images, point clouds, depth maps and camera poses.

By Hilton Raj, Vishnuram AV

llms rag agents robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 2

An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation

arXiv:2606. 00987v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have shown strong visual understanding and language-guided grounding abilities, yet their capacity for multi-temporal visual reasoning remains underexplored.

By Bingyu Li, Da Zhang, Tao Huo, Zhiyuan Zhao, Junyu Gao, Xuelong Li

llms agents computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jun 2

CARES: Context-Aware Resolution Selector for VLMs

arXiv:2510. 19496v3 Announce Type: replace-cross Abstract: Large vision-language models (VLMs) commonly process images at native or high resolution to remain effective across tasks.

By Moshe Kimhi, Nimrod Shabtay, Raja Giryes, Chaim Baskin, Eli Schwartz

llms multimodal benchmarks

arXiv AI

Jun 2

JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions

arXiv:2606. 01703v1 Announce Type: cross Abstract: We address the challenge of generating high-fidelity, long-form soundtracks that remain coherent across scene transitions.

By Jiashuo Yu, Yao Yao, Boyu Chen, Alex Wang

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

THRD: A Training-Free Multi-Turn Defense Framework for Jailbreak Attacks on Large Language Models

arXiv:2606. 01738v1 Announce Type: cross Abstract: Multi-turn jailbreak attacks pose a growing threat to LLMs by exploiting conversational dynamics such as gradual escalation and cross-turn coordination.

By Zhiqing Ma, Zhonghao Xu, Dong Yu, Chen Kang, Changliang Li, Pengyuan Liu

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision

arXiv:2606. 01992v1 Announce Type: cross Abstract: Industrial anomaly detection has historically been a unimodal task.

By Stefano Samele, Eugenio Lomurno, Teodora Jovanovic, Sanjay Shivakumar Manohar, Alberto Crivellaro, Matteo Matteucci

llms rag multimodal benchmarks

arXiv AI

Jun 2

DeepIPCv3: Event-Aware Multi-Modal Sensor Fusion for Sudden Pedestrian Crossing Avoidance

arXiv:2606. 01277v1 Announce Type: cross Abstract: Current end-to-end autonomous driving systems predominantly rely on frame-based sensors, which suffer from inherent perception latency and motion blur during highly dynamic encounters, specifically sudden pedestrian crossings.

By Oskar Natan, Andi Dharmawan, Aufaclav Zatu Kusuma Frisky, Jazi Eko Istiyanto, Jun Miura

llms agents robotics multimodal benchmarks safety

arXiv AI

Jun 2

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

arXiv:2606. 01057v1 Announce Type: cross Abstract: Procedural 3D modeling through code is emerging as a versatile paradigm, offering deterministic, engine-ready, and precisely editable assets that neural 3D generators inherently lack.

By Yipeng Gao, Lei Shu, Genzhi Ye, Xi Xiong, Ameesh Makadia, Meiqi Guo, Laurent Itti, Jindong Chen

llms agents multimodal benchmarks

arXiv AI

Jun 2

CAFOSat: A Strongly Annotated Dataset for Infrastructure-Aware CAFO Mapping Using High-Resolution Imagery

arXiv:2606. 00548v1 Announce Type: cross Abstract: Concentrated Animal Feeding Operations (CAFOs) play an important role in agricultural production but are also associated with environmental, public health, and disease surveillance concerns.

By Oishee Bintey Hoque, Nibir Chandra Mandal, Mandy L Wilson, Samarth Swarup, Madhav Marathe, Abhijin Adiga

llms multimodal benchmarks

arXiv AI

Jun 2

Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models

arXiv:2606. 00275v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have demonstrated impressive performance on multimodal tasks through scaled architectures and extensive training.

By Zijie Zhou, Dandan Zhu, Hangxiangpan Wang, Heng Zhang, Huishen Jiao, Yi Zhao

llms multimodal safety

arXiv AI

Jun 2

Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion

arXiv:2606. 00299v1 Announce Type: cross Abstract: While Video Diffusion Models (VDMs) excel at synthesizing high-fidelity videos, enabling precise camera and scene control remains challenging.

By Jiayi Wu, Haoming Cai, Cornelia Fermuller, Christopher Metzler, Yiannis Aloimonos

diffusion fine-tuning multimodal

Get the daily briefing