The paper introduces a joint fact‑verification score that evaluates both answers and the evidence submitted with them. On the FEVEROUS dataset, replacing the DCUF evidence with UnifEE evidence improves the strict score by about 9.6 percentage points, while answer accuracy rises only 1.96 points. The study also shows that increasing context length for large language models yields modest evidence‑gain improvements, and that detailed answer‑evidence analyses uncover patterns missed by aggregate metrics.
arXiv:2605. 03534v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) grounds answers in retrieved passages, yet relevance does not guarantee sufficiency: a topical passage may still fail to justify the answer.
arXiv:2606. 23915v1 Announce Type: cross Abstract: Practice often treats automatic metrics for attribution in LLM retrieval-augmented generation as interchangeable.
arXiv:2607. 26929v1 Announce Type: cross Abstract: The same diagnostic result can support or challenge one causal claim yet fail to address another when the claims concern different populations, outcomes, estimands, pathways, or identifying assumptions.
EviScope is a new paired counterfactual benchmark that evaluates grounded language models by fixing the question while manipulating evidence—adding, removing, distracting, or contradicting it. The v1.1 dataset includes 40 four‑condition quartets with repaired counterfactual claims and span‑level support labels for automated assessment. Experiments on Qwen2.5‑7B, Llama 3.1 8B, and Gemini 3.5 Flash show that paired metrics reveal grounding behaviors hidden by simple answer accuracy, such as unsupported answers, conflict blindness, and incorrect non‑answer actions.
The paper introduces TrustSwap, a counterfactual test that swaps or removes source reliability labels while keeping evidence text constant, to evaluate how retrieval‑augmented fact‑checking models respond across verdict, confidence, and search decisions. Experiments on untrained and RL‑trained models show that confidence and search largely follow labels, yet label changes can flip a significant portion of verdicts, especially in larger models. The authors propose trust‑swap augmentation (TSA) to mitigate this shortcut, demonstrating reduced verdict flip rates and maintained accuracy in several settings, though its effectiveness diminishes at larger model scales.
arXiv:2606. 15127v1 Announce Type: new Abstract: Reasoning models are increasingly used in settings where the final answer is not the only object of review: educational tools may show students intermediate steps, decision-support systems may require human oversight, and audit workflows may inspect traces for misleading or biased input.
MedFabric is a new benchmark for detecting word‑level medical fabrications, comprising 646 fabricated statements each paired with a ground‑truth passage that shares the same LLM authorship and nearly identical wording. The study shows that current detectors perform poorly—expert clinicians achieve only 53.3% macro‑F1 and no detector family surpasses 60% without gold evidence—highlighting that detection hinges on evidence correctness rather than subtlety of fabrication. The authors demonstrate that a retrieval‑confidence gate can substantially improve performance, raising macro‑F1 from 61% to 74%.
arXiv:2605. 27752v3 Announce Type: replace Abstract: Is verbalized confidence better calibrated than token likelihood?
arXiv:2607. 26102v1 Announce Type: cross Abstract: Mathematical chain of thought (CoT) evaluation is commonly reduced to whether the final answer matches a reference.
arXiv:2608.31016v1 Announce Type: cross Abstract: Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the n...
arXiv:2608. 08944v1 Announce Type: cross Abstract: A failed retrieval-augmented generation (RAG) answer can be consistent with several unseen responses to evidence repair.
The paper investigates when it is better to return an existing draft answer or revise it using retrieved evidence in retrieval‑augmented QA systems. By grading both the draft and its candidate revision with the same correctness judge, the authors define a paired effect called recoverability and train policies to predict it before revision. Experiments on 25,870 open‑domain questions show that a recoverability‑based scorer outperforms a draft‑correctness scorer across multiple Llama setups, improving accuracy–revision trade‑offs and closing a significant portion of the oracle gap, though it still applies harmful revisions in a substantial fraction of cases.