arXiv Computation and Language September 2, 2026 By Elitsa Yotkova, Violeta Kastreva, Petar Velkov, Hristo Boyanov, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Read the original on arXiv Computation and Language →

The paper introduces a semantic correctness taxonomy that categorizes open‑ended QA answers into eight ordered classes, distinguishing between correct, verbose, and hallucinated responses. It releases two datasets—CAP‑Correctness and CAP‑Statements—to support benchmark evaluation and NLI‑based training. The authors also propose CAP (Context‑Aware Precision), a reference‑based metric that scores question‑conditioned statements via bidirectional NLI and demonstrates superior performance under a monotonicity protocol.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Hugging Face Trending Papers

Sep 1

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surfac...

llms nlp benchmarks

arXiv Computation and Language

Oct 7

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked w...

By Jorma Valjakka, Juhani Kivim\"aki, Juha Myll\"ari, Jukka K. Nurminen

llms nlp benchmarks safety

arXiv AI

Jun 6

Improving Answer Extraction in Context-based Question Answering Systems Using LLMs

arXiv:2606. 06197v1 Announce Type: cross Abstract: Question answering (QA) systems have achieved notable progress with the advent of large language models (LLMs).

By Hafez Abdelghaffar, Ahmed Alansary, Ali Hamdi

llms nlp fine-tuning benchmarks

arXiv Computation and Language

3d ago

When Answer Evaluation Misleads Confidence Selection: Fact-Grounded Form Grading for Open-Ended QA

The paper examines how calibration rules for open‑ended question answering can be misled by the choice of correctness labels, noting that exact‑match labels can produce misleading confidence estimates. It introduces FactForm, a fact‑grounded form grader that combines reference matching with evidence retrieval, achieving lower error and higher AUROC than traditional graders on a large benchmark. FactForm’s design allows a single LLM call per answer and demonstrates stable performance across runs.

By Zhexi Feng, Bingrui Zhang, Wuxi Chen

llms nlp benchmarks

arXiv AI

Jul 21

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

arXiv:2604. 09497v2 Announce Type: replace-cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.

By Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, C\'eline Hudelot, Pierre Colombo

llms

arXiv AI

Jul 1

SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

arXiv:2504. 07385v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness.

By Sher Badshah, Ali Emami, Hassan Sajjad

llms agents benchmarks

One email a morning, machine-written

Related stories

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

Improving Answer Extraction in Context-based Question Answering Systems Using LLMs

When Answer Evaluation Misleads Confidence Selection: Fact-Grounded Form Grading for Open-Ended QA

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA