← Back to all news
arXiv Computer Vision By Dieuwertje Alblas, Alma M. Liezenga, Jan Erik van Woerden, Fedor Taggenbrock, Dalia Aljawaheri, Klamer Schutte

Beyond Benchmarks: Using VLMs to Reveal Systematic Classification Failures Under Real World Conditions

Read the original on arXiv Computer Vision →

The paper investigates using Vision Language Models (VLMs) to accelerate verification and validation (V&V) of classification models by automatically detecting systematic errors. It introduces a VLM-based error slice detection (ESD) method that groups and labels errors, demonstrating its ability to identify perturbations in a non-military dataset and to cluster images by surroundings in a military context. The study highlights challenges such as underrepresentation of defence data in VLM training and limited contextual diversity, and suggests that while fully automated V&V is not yet feasible, VLMs could speed up the process in the future.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Oct 2

Domain generalization and synthetic data in object detection: the enabler, the probe, and the gap

The paper reviews domain generalization (DG) for object detection, highlighting its scarcity and unique challenges such as localization and multi-scale representation. It examines synthetic data from three angles: as an enabler that diversifies and aligns training data, as a probe that allows controlled experiments to uncover failure modes, and as a source of a synthetic‑to‑real gap that hampers deployment on real imagery. The authors conclude that future DG research must develop representation‑aware methods that explicitly address both localization and classification under domain shift.

By Elfi I. S. Hofmeijer, Ella P. Fokkinga, Friso G. Heslinga, Klamer Schutte, J\"orgen M. Karlholm
computer-visionsafety
More like this →
arXiv AI
Aug 19

Comparative Study of Out-of-the-Box Technology for Automatic Target Detection and Recognition

The paper evaluates out‑of‑the‑box object detection models for automatic target detection and recognition (ATD/R) in military settings. Six YOLO variants and two DETR variants were benchmarked on a new military dataset featuring vehicles, occlusions, and small targets, with performance measured in mAP@0.5 and mAP@0.5:0.95 across air‑to‑ground and ground‑to‑ground perspectives. Findings show larger models and DETR-based approaches perform best, fine‑tuning on the VisDrone dataset improves air‑to‑ground and small‑object performance, yet all models still struggle with small targets in air‑to‑ground scenarios.

By Alma M. Liezenga, Lotte Nijskens, Henrik R. Baumann, Stefan Becker, Simon Bensberg, Niccol\`o Camarlinghi, H{\aa}vard R. Eiring, Alexander W. Johnsgaard, Tanel Liiv, Giuseppe Martino, Matteo Marturini, Matthias Rapp, Jan Erik van Woerden, Alexander Wolpert, Hugo J. Kuijf
agentscomputer-visionfine-tuningbenchmarks
More like this →
arXiv AI
Sep 21

DiaVLo: Diagnosing Behaviours of Vision-Language Models

DiaVLo is a diagnostic framework for vision‑language models (VLMs) that uses human curation and VLM generation to create specifications of desired and observed behaviours, revealing potential misalignments. It also offers causal estimates to pinpoint the most influential concepts driving VLM behaviour. Experiments on several open‑source VLMs under classification and generation tasks show that DiaVLo’s behaviour labels correlate with model performance and illuminate how VLMs perceive, organise, and prioritise concepts.

By Lorenzo Corti, Jie Yang
llmsmultimodal
More like this →