← Back to all news
arXiv AI By Ancil Crayton

Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation

Read the original on arXiv AI →

The paper argues that relying solely on overall accuracy or F1 scores to validate automatic scorers in LLM evaluation can conceal prompt‑specific errors that distort comparative results. By auditing six models on 180 invented‑entity questions across six prompts, the authors show that a scorer’s overall F1 of .916 masks substantial variation—up to 33.3%—in error rates depending on the prompt. In a second audit, differences in measured prompt effects dropped dramatically when multiple raters were used, highlighting the need to validate the exact comparison rather than the scorer alone.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Aug 14

A Probe Direction Is a Property of Its Prompt

arXiv:2608. 13329v1 Announce Type: new Abstract: A model that behaves differently when it senses it is being tested would undermine the evaluations we rely on, so recent work has sought to read that sense directly from a model's activations.

By Valentin No\"el
More like this →
arXiv AI
Oct 21

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

The study compares single-agent and multi-agent large language model (LLM) setups for reviewing credit assessment memos against specified requirements. In a controlled experiment with 300 synthetic memos, a single agent examined six requirements in one call, while a team of three agents each checked two requirements. Results show the team identified 0.08 to 0.10 more defects on certain models but also produced more false alarms, used 2.2 to 2.4 times more tokens, and had mixed effects on balanced accuracy depending on instruction and model.

By Pei Liu
llmsagents
More like this →
arXiv Computation and Language
Sep 24

What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs

The paper introduces a joint fact‑verification score that evaluates both answers and the evidence submitted with them. On the FEVEROUS dataset, replacing the DCUF evidence with UnifEE evidence improves the strict score by about 9.6 percentage points, while answer accuracy rises only 1.96 points. The study also shows that increasing context length for large language models yields modest evidence‑gain improvements, and that detailed answer‑evidence analyses uncover patterns missed by aggregate metrics.

By Han Chen, Yingrui Li
llms
More like this →