Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computer Vision
Oct 23

What Do Verifiable Rewards Teach Video-Language Models About Time? A Controlled Multi-Model Study

The paper investigates how reinforcement learning from verifiable rewards (RLVR) affects video‑language models’ understanding of time. Four open models were fine‑tuned under different data recipes—verified synthetic questions, unverified self‑supervised tasks, a 1:1 mix, and a verified+real mix—and evaluated on in‑domain and out‑of‑domain real‑video benchmarks with frames ordered, shuffled, or absent. Results show that verified training boosts in‑domain accuracy, largely through non‑visual gains, but can severely harm out‑of‑domain performance unless real verified data is added; moreover, no recipe induces genuine temporal‑order grounding, indicating that verifiable rewards improve benchmark scores without teaching temporal understanding.

By Avyay Sadhu, Patrick Cooper
arXiv Computer Vision
Oct 23

VisCurate: Evaluating Behavioral Equivalence of Visual Skills

VisCurate is a framework and benchmark that curates visual skill libraries by evaluating the actual visual behavior of skills rather than relying on textual metadata. It runs candidate skill pairs on a shared probe battery, assigns one of six behavioral relations, and uses these to safely edit the library. The benchmark introduces controlled defect types, producing 300 reproducible instances, and demonstrates that VisCurate outperforms text-based baselines in detecting false merges and recovering mergeable pairs, while revealing high redundancy in real libraries.

By Md Farhan Ishraq, Md Farhan Ishmam, Reaz Hassan Joarder, Md Hasanul Kabir
arXiv Machine Learning
Oct 23

Prediction Uncertainty Controlled Computer Model Calibration

The paper introduces a two‑layer framework for computer‑model calibration that directly controls predictive miscoverage. First, a conditional flow‑matching model learns an amortized posterior for calibration parameters using simulator outputs perturbed with Gaussian‑process discrepancies. Second, conformal risk control arranges posterior draws into nested candidate sets and selects the smallest set whose fixed‑rank pointwise bands achieve a prescribed time‑averaged miscoverage on a synthetic calibration sample. The method yields bands that are about 20% the width of traditional KOH‑MCMC bands while maintaining near‑nominal miscoverage, and when applied to an age‑structured SEIR model of the 2010 Malawi measles outbreak, the bands are sharper and highlight structural inadequacies in the mechanistic model.

By Ryan Heejoon Byun, Dayeon Yoon, Won Chang
arXiv Computation and Language
Oct 23

CARTE: A Benchmark for Mapping Language Model Knowledge Across France

CARTE (Culturally Anchored Regional-Territorial Evaluation) is a multiple-choice benchmark designed to assess large language models’ ability to distinguish fine-grained, geographically grounded knowledge within France. It contains 3,132 questions covering 13 metropolitan regions across 12 thematic domains such as culture, language, demographics, economy, environment, and mobility, and includes a subset called CARTE‑LV for linguistic variation. The benchmark was validated through a four-stage process and used to evaluate 32 open-weight LLMs and three API models in zero-, one-, and three-shot settings, revealing performance disparities across regions and model scales.

By Sarah Almeida Carneiro (X), Christos Xypolopoulos (X, NTUA), Xiao Fei (X), Yang Zhang (X), Michalis Vazirgiannis (X, MBZUAI)
arXiv Computation and Language
Oct 23

RiskCueBench: Benchmarking Anticipatory Reasoning from Early Risk Cues in Video-Language Models

RiskCueBench is a new video‑understanding benchmark that focuses on early risk cues in video‑language models. The dataset annotates the earliest moment in a video that signals a potential safety concern, rather than providing the full sequence that includes the accident. Experiments show a notable gap in current systems’ ability to interpret evolving situations and anticipate future risky events from these early visual signals.

By Sha Luo, Yogesh Prabhu, Timothy Ossowski, Kaiping Chen, Junjie Hu
arXiv Computer Vision
Oct 23

ParaLin++: Accelerating Parallel Diffusion Integrator via Intrinsic Partially Linear Structure

ParaLin++ is a training‑free, step‑parity parallel sampler that accelerates diffusion model sampling by preserving the full‑step discretization while removing the solver’s temporal dependency. The authors first review quality metrics for training‑free sampling and show that many step‑shortening methods lose quality on semantic‑composition and long‑text benchmarks, whereas parallel sampling remains near‑lossless. Experiments on Stable Diffusion‑v2, LSUN Church, FLUX.1‑dev, and Ming‑Flash‑Omni 2.0 demonstrate speedups of 2.6×–4.8×, 1.68×, and 1.21× respectively while maintaining the full‑step Euler quality profile.

By Jianrong Lu, Zhiyu Zhu, Hui Liu, Junhui Hou
arXiv Computer Vision
Oct 23

Squaring-the-Sphere: Distortion-Free and Seamless Panorama Generation via CubeAtlas

Squaring-the-Sphere introduces CubeAtlas, a new panoramic representation that unifies all cubemap faces into a single canvas, preserving perspective geometry while enabling seamless panorama assembly. The framework leverages a pretrained diffusion transformer with a layout‑aware sparse attention mask to reduce irrelevant token interactions, achieving higher quality than equirectangular projection with lower computational cost. Additionally, a training‑free super‑resolution and fusion scheme enhances fine details before converting to a final ERP panorama, resulting in improved geometric fidelity, cross‑view consistency, and high‑resolution visual quality across benchmarks.

By Xuhui Zuo, Chenjie Cao, Tengfei Wang, Junta Wu, Yisu Zhang, Chunchao Guo
arXiv Computation and Language
Oct 23

TT-SAX: A Two-Track Time Series Symbolization for Natural Language Shape Querying

The paper introduces TT‑SAX, a two‑track time series symbolization designed specifically for natural language shape querying. It addresses three identified shortcomings of existing symbolizations—missing direction, too fine a scale for direction, and arbitrary letters—by adding trend letters measured across neighboring segments and aligning them with query adjectives. Experiments on the NLQTSBench benchmark show TT‑SAX outperforming six classical symbolizations across six large language models, with ablation studies confirming the contribution of each design choice.

By Zhao Tan, Xianglong Bao, Zhipeng Xi, Ming Jin
arXiv Computer Vision
Oct 23

WildFakeToolBench: Benchmarking AI-Generated Image Detectors with Web-Sourced Image Processing

WildFakeToolBench is a benchmark suite comprising 156 executable image‑processing tools sourced from 220 public repositories across 142 domains. The suite evaluates six state‑of‑the‑art AI‑generated image detectors on 105 images processed by these tools, revealing that some tools can evade detectors with success rates up to 100%. The study demonstrates that web‑sourced tools pose significant in‑the‑wild risks and underscores the need for robust detector evaluation frameworks.

By Mingjie Li, Yukun Jiang, Yicong Tan, Yang Zhang
arXiv Computer Vision
Oct 23

ReCoG: Relational Concept Graph for Training-Free Personalization

ReCoG introduces a Relational Concept Graph that enhances training‑free personalization of vision‑language models by representing concepts as nodes linked with directed edges capturing discriminative cues between concept pairs. During inference, the graph retrieves a shortlist and resolves ambiguity through explicit pairwise comparisons guided by these relational edges, ensuring the selected concept consistently outperforms competitors. The method demonstrates significant improvements on the new FINGER benchmark for confusable concept sets and also outperforms prior approaches on established personalization tasks such as captioning and VQA.

By Changseon Yu, Hyeokjun Kweon
arXiv Computer Vision
Oct 23

Stream4D: Streaming 4D Reconstruction with Training-free Motion Decoupling via Causal KV-Cache Attention

Stream4D is a training‑free, motion‑decoupled streaming 4D reconstruction framework that can be integrated into causal Transformer backbones such as StreamVGGT and Stream3R. It exploits the causal attention’s KV cache to aggregate temporal motion cues, fuses them with depth priors, and produces precise dynamic masks that separate static and dynamic regions. A soft key masking scheme then suppresses dynamic interference in attention, improving camera pose estimation and reducing ghosting in long sequences.

By Wei Zhang, Ying Feng, Mengtong Wu, Shiqiang Gong, Huaqi Zhang, Qi Zhang
arXiv Computer Vision
Oct 23

WorldForge V2: Eliciting Camera-Trajectory Alignment in Video Diffusion Models

WorldForge V2 introduces a training‑free guidance framework that refines video diffusion outputs along a target camera path using warped references and optical‑flow‑identified latent channels for selective guidance. It employs a dual‑path strategy to correct artifacts, a v‑cache to reduce refinement cost, and Trajectory LoRA adapters trained in two stages to focus on trajectory‑responsive network blocks. The approach achieves state‑of‑the‑art generation quality and trajectory accuracy with minimal training data, enabling reliable camera control across unseen scenes.

By Chenxi Song, Yanming Yang, Tong Zhao, Ruibo Li, Joey Tianyi Zhou, Chi Zhang
arXiv Computation and Language
Oct 23

A Sociolinguistic Analysis of Automatic Speech Recognition Bias in Newcastle English

The paper examines how automatic speech recognition (ASR) systems perform on Newcastle English, a regional dialect of North‑East England, using spontaneous speech from the Diachronic Electronic Corpus of Tyneside English. It analyzes over 3,000 transcription errors, classifying them by linguistic domain and social variables such as gender, age, and socioeconomic status, and highlights that phonological variation—especially vowel quality and glottalisation—drives most errors. The study finds that error rates differ across social groups, with men and speakers at age extremes experiencing higher rates, indicating that ASR errors are socially patterned rather than random.

By Dana Serditova, Kevin Tang
arXiv AI
Oct 23

From Reasoning to Action: Benchmarking Frontier AI Models on LIBERO-10 Robot Manipulation Tasks

The paper introduces a closed‑loop framework to evaluate frontier AI models on the LIBERO‑10 robot manipulation benchmark, allowing models to control a robot via visual inputs, proprioception, and tool calls without task‑specific fine‑tuning. Six models (GPT‑6 Astra, GPT‑6.1 Sol, GPT‑6 Luna, Claude Opus 5, Claude Opus 5.5, Gemini‑4 Argon) were tested across 18 configurations and 1,800 episodes, with GPT‑6 Astra achieving the highest success rate of 44%. Results show that higher reasoning effort generally improves performance, yet benefits vary by model and task, and many models struggle with precise spatial arrangements and complex multi‑step interactions.

By May Espinola, Qifan Zhang, Yu Xiang
arXiv Machine Learning
Oct 23

Conservative Meta-Agent Search: Architecture Adaptation under Non-Stationarity

The paper introduces Conservative Meta-Agent Search (CMAS), a method for adapting agent architectures in non‑stationary environments. CMAS retrieves archived designs, proposes typed graph edits, and evaluates them through paired shadow replay with variance‑penalized scoring and a simulated canary stage before promotion. Using the MetaInvest‑Bench simulator, CMAS reduces surrogate dynamic‑oracle regret and improves held‑out utility across 96 financial streams, while maintaining low false‑promotion rates and modest inference cost.

By Ali Atiah Alzahrani
arXiv Machine Learning
Oct 23

On the Token Collapse Phenomenon in Textual Graph Soft Prompting

The paper investigates graph soft prompting, where a graph neural network generates graph-specific tokens for large language model prompts. It identifies a problem called token collapse, in which these tokens become nearly identical across different inputs, rendering them ineffective at conveying graph information. The authors show that even simple constant prompts can match or outperform collapsed tokens, and propose a two‑stage method to produce more varied tokens while maintaining performance.

By Andrea Giuseppe Di Francesco, Rishabh Jain, Franco Maria Nardini, Nicola Tonellotto, Pietro Li\`o, Fabrizio Silvestri
arXiv Machine Learning
Oct 23

AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining

AQL‑Lite is a framework that enables efficient adaptation of multilingual large language models to Modern Standard Arabic by combining a quality‑first data curation pipeline with a partial‑parameter continual pretraining recipe. The approach trains RS‑LoRA adapters, the last eight transformer blocks, and the embedding matrix using layer‑wise learning rates, achieving significant benchmark gains while reducing training time and resource usage. In large‑scale experiments, AQL‑Lite recovers most of the performance of full‑parameter continual pretraining and outperforms adapter‑only baselines, demonstrating its practicality for Arabic LLM development.

By Houssam Eddine Boukhalfa, Mourad Bouache, Mourad Dahmane
arXiv Machine Learning
Oct 23

Soft Ontology Priors over Latent Concept Posteriors for Weakly Supervised Neuro-Symbolic Learning

The paper introduces a method that injects a soft ontology prior into the training objective of weakly supervised neuro-symbolic predictors, aiming to prevent reasoning shortcuts that make latent concepts unidentifiable. By compiling the prior with the task constraint into a smooth probabilistic circuit, the authors show that the resulting structured concept posterior can be efficiently computed and that noisy expert axioms only redistribute prior mass without eliminating valid concept support. Theoretical analysis demonstrates bounded log‑density perturbations under axiom noise, while experiments on a real‑world driving benchmark and controlled tasks validate the approach.

By Tengwei Song, Robert Hoehndorf
arXiv Machine Learning
Oct 23

EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence

EpiSEM-Bench is an executable benchmark designed to assess how language‑model agents handle missing safety‑relevant evidence. It comprises 300 matched unsafe/safe task pairs across six domains, systematically suppressing critical facts to measure unsafe execution and failure cardinality. An independent audit tool evaluates whether agents seek missing facts and withhold prohibited actions while completing permissible tasks, revealing variation among ten models in omission robustness, evidence acquisition, and appropriate action.

By Qijin Zhang, Hanzhen Wang, Yuehan Zhao, Haoyu Dai, Yao Shu, Zhongxiang Dai, Shuang Qiu
arXiv Machine Learning
Oct 23

FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers

FLAWS is a new benchmark that tests large language models’ ability to find and pinpoint errors that invalidate key claims in scientific papers. The dataset contains 713 paper‑error pairs created by inserting claim‑invalidating mistakes into peer‑reviewed articles using LLMs, and it includes an automated metric for measuring error detection and localization. Five state‑of‑the‑art LLMs were evaluated, with GPT‑5 achieving the highest accuracy of 39.1% when selecting the top 10 candidate error texts.

By Sarina Xi, Vishisht Rao, Justin Payan, Nihar B. Shah