Relevance Is Not Sufficient Evidence: Detecting Evidence Gaps Before Generation in RAG
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The paper investigates whether incorporating an evidence-support signal into retrieval evaluation for retrieval‑augmented generation (RAG) improves downstream decision‑making. Across multiple benchmarks and a TREC RAG 2025 setting, the evidence signal alters retriever rankings but its benefits vary: it does not consistently enhance retriever training, its usefulness for system selection depends on generator instructions, and it does not reliably predict answer quality on unseen topics. Human filtering of evidence‑rich passages preserves useful content, yet evaluators disagree on whether this improves final answers, indicating that evidence‑aware evaluation alone does not guarantee better downstream outcomes.
The paper introduces RAG‑Stress, a diagnostic protocol that tests how retrieval‑augmented generation models handle misleading evidence. By fixing the question and reference answer, editing a single assertion to support an incorrect answer, and varying source‑priority policies and evidence positions, the study measures misleading rate (MR) and clean accuracy across fifteen systems on multiple QA datasets. Results show that prioritizing documents increases MR by 10.9–13.5 percentage points, with the highest MR when the misleading span appears at the end of the evidence.
arXiv:2607. 25600v1 Announce Type: cross Abstract: Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation.
The paper introduces the concept of LLM‑specific utility, defining it as the performance gain a target large language model (LLM) achieves when provided with a passage compared to answering without evidence. A benchmark of utilitarian passages is built for four LLMs (Qwen3‑8B/14B/32B and Llama 3.1‑8B) across three QA datasets, revealing that each model benefits most from its own tailored evidence and that evidence optimized for other models is consistently suboptimal. The authors also create SpecUBench, a benchmark for LLM‑specific utility judgment, and show that current utility‑aware retrieval methods largely capture model‑agnostic usefulness, struggling to estimate LLM‑specific utility. "whyItMatters":"The study demonstrates that retrieval‑augmented generation must consider model‑specific evidence selection to truly improve LLM performance, highlighting a gap in existing utility‑aware methods."
arXiv:2610.26951v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) is the dominant paradigm for grounding large language models in external corpora, but its similarity-driven, chunk...
Retrieval-augmented generation improves knowledge-intensive question answering, but indiscriminate retrieval can introduce irrelevant evidence and unnecessary computation. We investigate whether verbalized confidence from black-box language models can serve as an actionable signal for retrieval routing.