← Back to all news
arXiv Computation and Language By Zhexi Feng, Bingrui Zhang, Wuxi Chen

When Answer Evaluation Misleads Confidence Selection: Fact-Grounded Form Grading for Open-Ended QA

Read the original on arXiv Computation and Language →

The paper examines how calibration rules for open‑ended question answering can be misled by the choice of correctness labels, noting that exact‑match labels can produce misleading confidence estimates. It introduces FactForm, a fact‑grounded form grader that combines reference matching with evidence retrieval, achieving lower error and higher AUROC than traditional graders on a large benchmark. FactForm’s design allows a single LLM call per answer and demonstrates stable performance across runs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Sep 2

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

The paper introduces a semantic correctness taxonomy that categorizes open‑ended QA answers into eight ordered classes, distinguishing between correct, verbose, and hallucinated responses. It releases two datasets—CAP‑Correctness and CAP‑Statements—to support benchmark evaluation and NLI‑based training. The authors also propose CAP (Context‑Aware Precision), a reference‑based metric that scores question‑conditioned statements via bidirectional NLI and demonstrates superior performance under a monotonicity protocol.

By Elitsa Yotkova, Violeta Kastreva, Petar Velkov, Hristo Boyanov, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov
llmsnlpbenchmarks
More like this →
arXiv Computation and Language
Aug 27

Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking

The paper demonstrates that open‑ended Theory‑of‑Mind trackers can produce valid beliefs that are absent from finite reference sets, and that treating unmatched outputs as false can reverse model‑selection rankings. By recoding references for 259 beliefs, the authors show a dramatic drop in weighted prevalence and a reversal of strictly proper Brier risk, with similar distortions observed in a 301‑question NQ‑open DPR‑BERT pipeline. The study further reveals that 90‑96% of audited unmatched beliefs are literally true, and introduces a TriSource‑Restore method that anchors reference labels to a probability‑sampled human pilot to restore calibration and ranking integrity.

By Zhexi Feng, Wuxi Chen, Bingrui Zhang
More like this →