arXiv Computation and Language

Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

The paper introduces a user‑centric audit framework that separates the users who create benchmark content from those whose needs the evaluation results are meant to represent. It operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures, and applies the framework to 16 resources. Findings show that American‑centered conventions dominate many text benchmarks, geographic visibility in image‑text resources correlates strongly with national GDP, and documentation reveals a gap between source‑user and target‑user contexts. Paired scoring experiments demonstrate that meaning‑preserving answer variants can drastically lower accuracy metrics, highlighting the sensitivity of evaluation scores to answer phrasing.

arXiv Computation and Language

Oct 9

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

CiteVQA is a benchmark that evaluates multimodal large language models on both answering document questions and providing element‑level bounding‑box citations for the evidence used. It contains 1,897 questions across 711 PDFs in seven domains and two languages, with ground‑truth citations generated via an automated pipeline and strict quality control. The evaluation metric, Strict Attributed Accuracy (SAA), rewards a prediction only when both the answer and the cited region are correct, revealing widespread attribution hallucination in current models.

By Dongsheng Ma, Jiayu Li, Zhengren Wang, Yijie Wang, Jiahao Kong, Weijun Zeng, Jutao Xiao, Jie Yang, Bangrui Xu, Yuhan Wang, Bin Wang, Conghui He

llms multimodal benchmarks safety

arXiv AI

Aug 3

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

arXiv:2607. 28801v1 Announce Type: cross Abstract: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.

By Philipp D. Siedler, Jordan Sassoon

llms agents benchmarks safety

arXiv Computation and Language

1d ago

Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking

arXiv:2604.13776v3 Announce Type: replace-cross Abstract: Watermarking is becoming the default mechanism for AI content authentication, with governance policies and frameworks referencing it as infra...

By Alexander Nemecek, Osama Zafar, Yuqiao Xu, Wenbiao Li, Erman Ayday

benchmarks safety

arXiv AI

Sep 15

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

NoteVQA is a new benchmark that collects 252 real‑life visual questions from the Chinese image‑sharing platform Xiaohongshu, covering 12 topics and 7 user intents. Each question is paired with a concise expert reference and a human‑audited interleaved answer that blends text and visual evidence. The study evaluates VLMs on short‑answer correctness and interleaved answer quality using a new AgenticInterleave framework and a 12‑dimensional IVR‑12 rubric, finding that even state‑of‑the‑art models achieve only about 53% accuracy and lag behind human references in content quality.

By Haonan Jiang, Guojian Zhan, Jiancong Xie, Shijun Wan, Dongiia Zhao, Cheng Chen, Yahui Liu, Yao Hu, Chuan Mu

llms agents multimodal benchmarks

arXiv AI

Jul 21

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

arXiv:2604. 09497v2 Announce Type: replace-cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.

By Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, C\'eline Hudelot, Pierre Colombo

llms

arXiv Computation and Language

1d ago

Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval

arXiv:2610.37162v1 Announce Type: cross Abstract: Coverage-evaluation benchmarks quantify whether retrieved evidence covers the different pieces of information needed to satisfy an information need....

By Pasha Abrishamchian, Eugene Yang, Andrew Yates, Nazli Goharian, Ophir Frieder

llms benchmarks

arXiv AI

Sep 10

AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents

AtomCite is an agentic framework that verifies and corrects page‑level citations in multi‑page documents by parsing answers into claims, checking each claim against the cited page image, and applying a deterministic repair policy. The authors introduce DocCite, the first benchmark for this task, built on MP‑DocVQA and DUDE, containing 928 injected instances and 1,909 verified natural errors. Across Gemini, Claude, and GPT models, AtomCite achieves about 93% verification accuracy and improves citation precision from 34% to 87‑90%, while also enhancing hallucination detection in open‑source models.

By Chen Qian, Yimeng Wang, Yu Chen, Lingfei Wu, Andreas Stathopoulos

llms agents computer-vision benchmarks safety

Hugging Face Trending Papers

Aug 2

Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

Reference-based text evaluation metrics, which are widely used to assess natural language generation systems, score a candidate response by comparing it with a reference response. The reliability of an evaluation metric is usually judged by its statistical correlation with human ratings.

llms agents nlp robotics benchmarks safety

arXiv AI

Oct 15

VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

arXiv:2610.16441v1 Announce Type: cross Abstract: Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Exp...

By Subhadeep Roy, Christoph Leiter, Steffen Eger

llms fine-tuning multimodal benchmarks

arXiv AI

Sep 10

A Hierarchical Consistency Framework for Auditing Retrieval-Augmented Generation Systems

arXiv:2609.07075v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) is commonly evaluated by whether the final answer is correct. That test is insufficient: an answer can match its r...

By Ramon Gonzalez, Antonio Diaz

rag

Hugging Face Trending Papers

Jul 29

SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) methods are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers.

llms agents multimodal benchmarks safety

arXiv Machine Learning

Aug 4

Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

arXiv:2608. 01423v1 Announce Type: cross Abstract: Reference-based text evaluation metrics, which are widely used to assess natural language generation systems, score a candidate response by comparing it with a reference response.

By Shengwei Xu, Yuxuan Lu, Yifan Wu, Jason Hartline, Grant Schoenebeck

llms agents nlp robotics benchmarks safety

Related stories