← Back to all news

Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Computation and Language
Sep 23

Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining

The paper introduces WaterBERT, a domain‑adapted encoder model trained on a 2.97‑billion‑token water treatment corpus to capture domain‑specific semantics for literature mining. Fine‑tuned versions of WaterBERT outperform general‑purpose and other domain BERT models on tasks such as treatment process classification, named entity recognition, and relation extraction. The authors also demonstrate WaterBERT’s utility in large‑scale processing, generating coherent research topics, building a structured knowledge graph from 693,211 abstracts, and creating a Water Knowledge‑Enhanced Retrieval System that surpasses text‑based baselines.

By Mudi Zhai (UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales, Sydney, NSW 2052, Australia), Ruihong Qiu (School of Electrical Engineering and Computer Science, The University of Queensland, Brisbane, QLD 4072, Australia), Qingyun Zeng (Microsoft Copilot Studio AI, Redmond, WA 98052, United States, Departments of Mathematics & Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, United States), T. David Waite (UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales, Sydney, NSW 2052, Australia), Bing-Jie Ni (UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales, Sydney, NSW 2052, Australia), Haoran Duan (UNSW Water Research Centre, School of Civil and Environmental Engineering, The University of New South Wales, Sydney, NSW 2052, Australia, Department of Civil Engineering, The University of Hong Kong, Pokfulam, Hong Kong SAR, China)
llmsnlpfine-tuningbenchmarks
More like this →
arXiv Computation and Language
Sep 23

Same Chart, Different Story: Bias in Vision-Language Chart Interpretation

The paper introduces ChartBias, a benchmark of 820 real-world charts covering six social attributes, designed to audit bias in vision‑language models (VLMs) that interpret charts. Across 12 VLMs, the study identifies three failure modes—narrative shift, group hallucination, and preference polarity—where models produce different or misleading narratives when the referenced social group changes. A multi‑agent mitigation framework is proposed, separating evidence extraction from group‑conditioned generation and using a counterfactual judge, which reduces narrative shift while maintaining chart‑grounded reasoning.

By Mizanur Rahman, Huan Wu, Arash Asgari, Enamul Hoque Prince, Laleh Seyyed-Kalantari
llmsagentsnlpmultimodalbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 23

Quantitative Evidence Mining for Plausibility-Aware Biomedical AI: A Narrative Review and Conceptual Framework

The article proposes a framework called quantitative evidence mining to transform biomedical findings into structured, context-rich evidence units. It outlines core elements such as claim, measured entity, value, comparator, population, conditions, temporal context, uncertainty, provenance, validation, and expert review. The authors present an eight-stage reference architecture and emphasize that plausibility should remain multidimensional rather than collapsed into a single truth label, linking extraction to evidence synthesis for applications like clinical trials, biomarker research, and knowledge-graph construction.

By Negin Sadat Babaiha, Stefan Geissler, Marie-Christine Simon, Martin Hofmann-Apitius, Marc Jacobs
multimodalbenchmarks
More like this →
arXiv Computation and Language
Sep 23

CONCAT: Consensus- and Confidence-Driven Ad Hoc Teaming for Efficient LLM-Based Multi-Agent Systems

CONCAT is a training‑free framework that improves the efficiency of large language model (LLM) based multi‑agent systems by clustering agents according to their initial answers and selecting cluster leaders based on confidence. It uses a Theory‑of‑Mind‑inspired heuristic to predict collaboration benefits between leaders, then prunes communications to form an ad‑hoc network that reduces latency. Experiments on three LLMs and benchmarks show up to 2.02× higher accuracy/latency ratio than LLM‑Debate and a 50.1% latency reduction on Qwen2.5‑14B‑Instruct without task‑specific training.

By Ziyang Ma, Dingyi Zhang, Sichu Liang, Jiajia Chu, Pengfei Xia, Hui Zang, Deyu Zhou
llmsagentsfine-tuningbenchmarks
More like this →
arXiv Computer Vision
Sep 23

Annual Earth-observation embeddings encode wildfire disturbance and support simplified burned area mapping

Annual Earth‑observation embeddings, specifically Tessera, can encode wildfire disturbance signals well enough to map burned areas without needing curated fire‑specific imagery or dense time‑series analysis. In tests, linear models using a single Tessera embedding matched or outperformed paired pre‑ and post‑fire HLS imagery and post‑fire imagery alone, achieving high F1 scores for burn‑scar delineation and regional mapping. The approach successfully mapped all same‑year fires in benchmark scenes, recovered 97% of California burned area without California training data, transferred to European fires with high accuracy, and even estimated ignition timing within a 13‑day error margin. "whyItMatters":"The study demonstrates that pre‑trained annual embeddings can simplify and scale burned‑area mapping, reducing reliance on dense time‑series data and enabling more efficient wildfire monitoring."

By Jovana Knezevic, Clement Atzberger, Zhengpeng Feng, Adam F. A. Pellegrini, Srinivasan Keshav, David Coomes
ragcomputer-visionbenchmarks
More like this →
arXiv Computer Vision
Sep 23

Metric-Bench: Exploring In-context Spatial Metric Reasoning in VLMs for Indoor Scenes

Metric-Bench introduces a new benchmark for Vision‑Language Models (VLMs) that focuses on metric‑spatial reasoning in indoor scenes by using in‑image reference objects with known dimensions. The accompanying MetricReasoner fine‑tuning recipe employs structured prompts and numerical rewards to implicitly learn 2D‑to‑3D mapping without camera intrinsics. Experiments show that this approach improves spatial metric understanding by 43.1 % over existing models and boosts downstream embodied tasks, while also delivering gains on general VLM benchmarks.

By Yuling Xi, Haokai Zhang, Muzhi Zhu, Hao Zhong, Zongze Du, Hengyu Zhao, Chenchen Jing, Yufei Yin, Bin Qin, Yongjie Yang, Zhenbo Luo, Hao Chen, Chunhua Shen
llmsagentsroboticsfine-tuningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 23

MIAR: Medical Image Super-Resolution With Autoregressive Modeling

MIAR introduces a multi‑scale autoregressive framework for medical image super‑resolution, treating the task as a conditional, progressive next‑scale prediction. It incorporates a Scale‑Adaptive Structural Decoder to preserve structural fidelity and uses a hierarchical beam search during inference to reduce recursive error accumulation. Experiments show MIAR outperforms existing methods, achieving a 7.86% MUSIQ improvement and a 2.02× speedup over diffusion‑based approaches.

By Fang Li, Yinglong Li, Hongyu Wu, Yang Gao, Minwei Zhao, Aimin Hao
diffusionefficiencybenchmarks
More like this →