The paper introduces a semantic correctness taxonomy that categorizes open‑ended QA answers into eight ordered classes, distinguishing between correct, verbose, and hallucinated responses. It releases two datasets—CAP‑Correctness and CAP‑Statements—to support benchmark evaluation and NLI‑based training. The authors also propose CAP (Context‑Aware Precision), a reference‑based metric that scores question‑conditioned statements via bidirectional NLI and demonstrates superior performance under a monotonicity protocol.
Reliable evaluation of open-ended question answering remains a bottleneck for measuring answer correctness of modern LLMs. Unlike multiple-choice tasks, free-form answers may be correct in many surfac...
arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked w...
arXiv:2606. 06197v1 Announce Type: cross Abstract: Question answering (QA) systems have achieved notable progress with the advent of large language models (LLMs).
The paper examines how calibration rules for open‑ended question answering can be misled by the choice of correctness labels, noting that exact‑match labels can produce misleading confidence estimates. It introduces FactForm, a fact‑grounded form grader that combines reference matching with evidence retrieval, achieving lower error and higher AUROC than traditional graders on a large benchmark. FactForm’s design allows a single LLM call per answer and demonstrates stable performance across runs.
arXiv:2604. 09497v2 Announce Type: replace-cross Abstract: Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases.
arXiv:2504. 07385v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness.
arXiv:2609.15561v1 Announce Type: new Abstract: Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy?...
The paper introduces Evidence Sufficiency Boundary Training, a framework that teaches models to abstain from answering until the supplied evidence is fully sufficient, and to remain stable when additional redundant evidence is added. By constructing ordered evidence chains from datasets such as HotpotQA, 2WikiMultiHopQA, and MuSiQue, the method applies level supervision, a boundary flip margin, post‑boundary stability, and answer recall protection. Experiments with Qwen2.5‑3B‑Instruct and LoRA adaptation show improved boundary localization (flip accuracy 0.807 vs 0.781) and a lower unsupported‑answer rate (0.095 vs 0.101) while maintaining competitive raw QA F1.
arXiv:2608. 10315v1 Announce Type: cross Abstract: Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching.
arXiv:2609.24799v1 Announce Type: new Abstract: Post-training quantization (PTQ) enables efficient deployment of large language models, and PTQ methods are usually optimized and evaluated with generi...
arXiv:2607. 06641v1 Announce Type: cross Abstract: Large language models (LLMs) achieve promising results on medical question answering benchmarks, yet their use in public health is constrained by hallucinations and the rapid evolution of official guidance.
arXiv:2607. 22554v1 Announce Type: new Abstract: Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same question is phrased in different but equivalent ways.