arXiv Computation and Language September 2, 2026 By Elitsa Yotkova, Violeta Kastreva, Petar Velkov, Hristo Boyanov, Dimitar Dimitrov, Ivan Koychev, Preslav Nakov

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Read the original on arXiv Computation and Language →

The paper introduces a semantic correctness taxonomy that categorizes open‑ended QA answers into eight ordered classes, distinguishing between correct, verbose, and hallucinated responses. It releases two datasets—CAP‑Correctness and CAP‑Statements—to support benchmark evaluation and NLI‑based training. The authors also propose CAP (Context‑Aware Precision), a reference‑based metric that scores question‑conditioned statements via bidirectional NLI and demonstrates superior performance under a monotonicity protocol.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Hugging Face Trending Papers

Sep 1

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surfac...

llms nlp benchmarks

arXiv AI

Jun 6

Improving Answer Extraction in Context-based Question Answering Systems Using LLMs

arXiv:2606. 06197v1 Announce Type: cross Abstract: Question answering (QA) systems have achieved notable progress with the advent of large language models (LLMs).

By Hafez Abdelghaffar, Ahmed Alansary, Ali Hamdi

llms nlp fine-tuning benchmarks

arXiv AI

Jul 21

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

arXiv:2604. 09497v2 Announce Type: replace-cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.

By Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, C\'eline Hudelot, Pierre Colombo

llms

arXiv AI

Jul 1

SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

arXiv:2504. 07385v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness.

By Sher Badshah, Ali Emami, Hassan Sajjad

llms agents benchmarks

arXiv Computation and Language

3d ago

Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation

arXiv:2609.15561v1 Announce Type: new Abstract: Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy?...

By Sarra Gharsallah, Adele Robaldo, Mariia Tokareva, Giovanni Gatti Pinheiro, Ilyana Guendouz, Rapha\"el Troncy, Paolo Papotti, Pietro Michiardi

llms

arXiv Computation and Language

Sep 3

Learning Evidence Sufficiency Boundaries for Selective Answering in Grounded Multi-Hop QA

The paper introduces Evidence Sufficiency Boundary Training, a framework that teaches models to abstain from answering until the supplied evidence is fully sufficient, and to remain stable when additional redundant evidence is added. By constructing ordered evidence chains from datasets such as HotpotQA, 2WikiMultiHopQA, and MuSiQue, the method applies level supervision, a boundary flip margin, post‑boundary stability, and answer recall protection. Experiments with Qwen2.5‑3B‑Instruct and LoRA adaptation show improved boundary localization (flip accuracy 0.807 vs 0.781) and a lower unsupported‑answer rate (0.095 vs 0.101) while maintaining competitive raw QA F1.

By Haruto Sato, Yuki Tanaka, Ren Nakamura, Aoi Kobayashi, Mei Ito

nlp fine-tuning

One email a morning, machine-written

Related stories

How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

Improving Answer Extraction in Context-based Question Answering Systems Using LLMs

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation

Learning Evidence Sufficiency Boundaries for Selective Answering in Grounded Multi-Hop QA