Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

The paper introduces a user‑centric audit framework that separates the users who create benchmark content from those whose needs the evaluation results are meant to represent. It operationalizes seven dimensions across benchmark content, metadata, documentation, and scoring procedures, and applies the framework to 16 resources. Findings show that American‑centered conventions dominate many text benchmarks, geographic visibility in image‑text resources correlates strongly with national GDP, and documentation reveals a gap between source‑user and target‑user contexts. Paired scoring experiments demonstrate that meaning‑preserving answer variants can drastically lower accuracy metrics, highlighting the sensitivity of evaluation scores to answer phrasing.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• multimodal
• benchmarks