← Back to all news

Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,506 stories · RSS feed

arXiv Computation and Language
Sep 1

World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning

The paper introduces a framework that combines world models, which generate concrete visual rollouts of possible futures, with multimodal large language models (MLLMs) that perform abstract reasoning. It proposes a controlled concrete reasoning approach and a new training method called Privileged‑Future On‑Policy Self‑Distillation (PF‑OPSD), which uses ground‑truth future videos as privileged teacher context during training while the student model never sees true futures at test time. Experiments on two human‑verified benchmarks, VRQABench and OpenWorldQA, show that PF‑OPSD improves performance by about 10–11% over baselines and enhances robustness to noisy or conflicting rollouts.

By Yucheng Zhou, Wei Tao, Yiwen Guo, Jianbing Shen
llmsefficiencymultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 1

Automated pipeline for herbarium label digitization

HERBIOME is a modular, end‑to‑end pipeline that automates the digitization of herbarium labels. It combines YOLOv8 for component detection, CRAFT Hezar for word‑level text localization, a fine‑tuned TrOCR model for mixed handwritten and printed text recognition, and GPT‑4o Mini for structuring metadata into standardized fields. Evaluation on 450 French specimens shows high surface similarity (MWS ≈ 0.616) and moderate semantic accuracy (SMA ≈ 0.442), with taxonomic fields identified as the main challenge.

By Hiba Abbad, Hanane Ariouat, Eva Perez Pimpare, Nicolas Turenne, Eric Chenin, Abderrazak Sebaa, Edi Prifti, Jean-Daniel Zucker, Youcef Sklab
llmsfine-tuningmultimodal
More like this →
arXiv Computer Vision
Sep 1

TopoAgent: A Structure-Aware Perception-to-Reasoning Framework for Diagram-to-Graph Topology Extraction with Large Vision-Language Models

The paper introduces TopoBench-180, a human‑verified benchmark of 180 structural diagrams with canonical graph annotations, and TopoAgent, a perception‑to‑reasoning framework that extracts graph topology from diagrams using large vision‑language models. TopoAgent combines grounded perception, global structural priors, node inventory construction, local‑to‑global relation reasoning, and consistency enforcement to progressively build the target graph. Experiments demonstrate that TopoAgent surpasses strong baselines, particularly in edge extraction, thereby advancing multimodal structured understanding for diagram‑to‑graph tasks.

By Bangwei Guo, Xujiang Zhao, Yanchi Liu, Wei Cheng, Shengyu Chen, Dongyue Li, Masaharu Morimoto, Takayuki Kuroda, Dimitris Metaxas, Haifeng Chen
llmsmultimodalbenchmarks
More like this →
arXiv Computation and Language
Sep 1

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

The study evaluates whether multimodal large language models (MLLMs) can produce open‑ended aesthetic critiques comparable to humans. Eight open‑weight MLLMs (7 B–397 B) and GPT‑5.5 were tested on 1,227 r/photocritique posts under various prompts, revealing that reference‑based similarity metrics often misrepresent model performance, while shorter critiques and image‑omission had limited impact. Human judges and annotators found the models’ critiques largely different from human ones, noting that models tend to be overly comprehensive and repetitive rather than selective and specific.

By Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie, Mahnoosh Alizadeh, Ramtin Pedarsani
llmsragmultimodal
More like this →
arXiv AI
Sep 1

Imag-Eval: a language-grounded framework for interpretable Text-to-Image instruction following evaluation

Imag‑Eval is a new language‑grounded benchmark for evaluating Text‑to‑Image models, focusing on how well they follow compositional natural‑language instructions. It disentangles prompt length from compositional difficulty by independently varying the number of instances and the combination of constraints (rules), providing 1,140 prompts and 8,842 rule combinations. The study shows that for structured skills, the difficulty is mainly driven by the number of grounded rules and their binding to instances rather than prompt length alone.

By Ibrahim Mohamed Serouis, David Jaramillo Duque
diffusionmultimodalbenchmarks
More like this →
arXiv AI
Sep 1

Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes

arXiv:2506.09557v2 Announce Type: replace-cross Abstract: While Multimodal Large Language Models (MLLMs) demonstrate impressive performance in benign scenarios, their cognitive reliability deteriorat...

By Zhaoyang Wei, Bowen Jiang, Xumeng Han, Jiashu Li, Xuehui Yu, Yuling Liu, Guorong Li, Zhenjun Han, Jianbin Jiao
llmsreinforcement-learningmultimodalbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 1

The Fragility of Jailbreak Robustness Across Operational States

The study shows that jailbreak robustness in language models is highly sensitive to operational-state changes. Even minor alterations to system prompts, not intended to affect safety, can dramatically shift attack success rates across seven aligned models and three jailbreak methods. The authors link these variations to changes in hidden representations along a refusal-related axis, which can predict jailbreak outcomes.

By Yuna Park, Hwang Youn Kim, Yujin Kim, Won Woo Ro, Suhyun Kim, Jae-In Hwang
llmsmultimodalsafety
More like this →
arXiv Computation and Language
Sep 1

SpanCalib-VLM: Calibrated Hallucination Span Detection in Vision-Language Models

SpanCalib-VLM is a hybrid system for detecting hallucinated text spans in Vision‑Language Models. It combines a multimodal sequence tagger (XLM‑RoBERTa‑Large + SigLIP) with a fine‑tuned generative VLM (Qwen3.5‑4B‑SHROOM‑SFT) and uses a Union‑Calibrated Fusion strategy to re‑score candidate spans. On the SHROOM‑Visions English evaluation split, the ensemble achieves a Pearson calibration correlation of 0.41, an overall IoU of 0.39, a clean‑response IoU of 0.91, and a detection accuracy of 70.7%.

By Amanuel Gizachew Abebe, Yasmin Moslem
llmsfine-tuningefficiencymultimodalsafety
More like this →
arXiv Computation and Language
Sep 1

Context-Aware Interleaved Batching for WhisperX

Context-Aware Interleaved Batching for WhisperX proposes a new batching strategy that combines WhisperX’s fast intra-audio batching with Whisper’s sequential context retention. By using voice activity detection (VAD) to define segment boundaries, the algorithm stabilizes Whisper’s text conditioning and preserves continuous historical context across batched audio segments. Experiments on long‑form audio benchmarks show reduced Word Error Rate and improved proper noun transcription while maintaining high‑throughput inference speeds.

By Carlos Bain, Max Bain
efficiencymultimodalbenchmarkssafety
More like this →