← Back to all news

Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,102 stories · RSS feed

arXiv AI
Sep 10

Concord: A Video Relational Algebra for Cross-Modal Query Optimization

Concord introduces a Video Relational Algebra (VRA) that models videos, transcripts, frames, and object tracks, enabling semantic video queries. It applies approximate optimizations to rewrite VRA queries, reducing large language model (MLLM) usage by processing transcripts or using detection and tracking instead of full-video MLLM joins. Experiments on soccer broadcasts and lectures show that Concord sends only a small fraction of video to the MLLM, cutting costs by up to 87%, and improves cross‑camera query accuracy from an F1 of .364 to .813 without any MLLM calls.

By Sultan Muratbek, Charisse Ivana Yeung, Chanwut Kittivorawong, Alvin Cheung
llmsmultimodal
More like this →
arXiv AI
Sep 10

GeoContext: One Context Ladder, Two Failure Modes in Vision-Language Geolocation: Flat Reliance on User-Provided Location Context and False Confirmation of Location Claims

GeoContext is a new vision‑language geolocation benchmark that introduces two tasks: GeoHint, where a model must localize an image given a coarse location hint, and GeoVerify, where a model must decide if an image was taken within 150 m of a claimed place. The benchmark builds a context ladder by stratifying nearby reference points by distance and referenceability, allowing the same image to be evaluated under varying context. Evaluation of five models on 109 sites in 30 cities shows that hint repetition is low, localization error grows with hint distance, and models struggle to achieve high discriminability in GeoVerify, with many false acceptances reported with high confidence.

By Yifan Zhang, Kai Wang
llmsmultimodalbenchmarkssafety
More like this →
arXiv AI
Sep 10

Geometry-Aware Test-Time Learning for Quantitative Spatial Reasoning

The paper introduces TTL‑SR, a geometry‑aware Test‑Time Learning framework designed to improve quantitative spatial reasoning in visual‑language models. By augmenting queries with geometrically coupled auxiliary prompts, filtering unreliable predictions, and updating models with a geometry‑aware multi‑objective loss on unlabeled test data, TTL‑SR adapts models to new domains without additional 3D supervision. Experiments show substantial accuracy gains on the Q‑Spatial‑ScanNet dataset for two state‑of‑the‑art VLMs.

By Gege Zhang, Shuaicheng Niu, Gang Dai, Lei Sun, Shuangping Huang
llmsmultimodal
More like this →
arXiv AI
Sep 10

Discovering Natural Transformation Vulnerabilities in Black-Box Vision Models

The paper introduces Adversarial Scenario Attack (ASA), a query‑based black‑box method that discovers natural transformation vulnerabilities in vision models by exploring background, weather, and material/color edits via a multimodal language model and a text‑guided generative editor. ASA outperforms previous query‑based generative attacks on ImageNet classifiers, achieving higher success rates with fewer queries while maintaining perceptual quality. The approach also shows image‑level and prompt‑level transferability, indicating reusable vulnerabilities across models and images.

By Dongsu Song, DaeYun GO, Jay Hoon Jung
llmsmultimodalsafety
More like this →
arXiv AI
Sep 10

MV-STRIDE: Enabling MLLMs to Master Multi-View Spatial Reasoning via Hierarchical Capability Modeling

MV-STRIDE is a Multi‑View hierarchical Spatial Reasoning dataset that models dependencies among perception, scene understanding, and contextual reasoning to support 3D spatial cognition. It introduces a QA generation pipeline that enforces cross‑view constraints, producing multi‑level reasoning tasks with chain‑of‑thought supervision. Experiments show that training on MV‑STRIDE yields state‑of‑the‑art performance on multi‑view spatial benchmarks, enabling MLLMs to reason robustly across diverse viewpoints.

By Jin Xu, Xiaojian Huang, Zhuodong Luo, Zhihong Zhang, Xin Liu, Jiansheng Wei, Xinzhi Wang, Jie Zhao, Xuejin Chen
llmsmultimodalbenchmarks
More like this →
arXiv AI
Sep 10

BlueprintAgent: Constraint-Triggered Targeted Revisits for Simulation-Ready Generation from Scanned Structural Blueprints

BlueprintAgent (BPA) is a multimodal agent that converts scanned reinforced‑concrete building blueprints into simulation‑ready frame models by treating a large language model (MLLM) as the primary reader and decision maker, while OCR and computer vision provide localized evidence. BPA implements engineering constraints as callable validators that trigger targeted MLLM revisits over specific regions, enabling precise beam‑column support, span count, and 3D continuity checks. In evaluations on 300 real scanned sheets from 20 projects, BPA achieved a macro‑averaged Beam F1 of 0.994, vastly outperforming single‑MLLM zero‑shot (0.301) and fixed‑pipeline (0.820) baselines.

By Zhouyuan Xu, Chen Yang, Linhao Wang, Jiansheng Fan, Chen Wang
llmsagentscomputer-visionmultimodalsafety
More like this →
arXiv AI
Sep 10

Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy

The paper investigates adding Greek language support to a Cosmos3 vision‑language‑action robot policy using only machine‑rephrased instructions and no architectural changes. It finds that many evaluation metrics can give misleading results, and that multilingual training with Greek yields a modest 6.7‑7.1 point advantage over a control, reaching about 40% of English performance. The study also shows that overfitting to a single translator’s phrasing can be mitigated by training on multiple phrasings, while warm‑starting from a language‑adapted world model or unfreezing the text tower actually harms performance.

By Ayoub Kirouane, Georgios Giaples, Christos Petrocheilos
roboticsmultimodalbenchmarks
More like this →