arXiv Computation and Language
Aug 28

Assessing the Downstream Utility of Evidence-Aware Retrieval in RAG

The paper investigates whether incorporating an evidence-support signal into retrieval evaluation for retrieval‑augmented generation (RAG) improves downstream decision‑making. Across multiple benchmarks and a TREC RAG 2025 setting, the evidence signal alters retriever rankings but its benefits vary: it does not consistently enhance retriever training, its usefulness for system selection depends on generator instructions, and it does not reliably predict answer quality on unseen topics. Human filtering of evidence‑rich passages preserves useful content, yet evaluators disagree on whether this improves final answers, indicating that evidence‑aware evaluation alone does not guarantee better downstream outcomes.

By Utshab Kumar Ghosh, Debayan Mukhopadhyay, Shubham Chatterjee
arXiv AI
Oct 9

RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation

The paper introduces RAG‑Stress, a diagnostic protocol that tests how retrieval‑augmented generation models handle misleading evidence. By fixing the question and reference answer, editing a single assertion to support an incorrect answer, and varying source‑priority policies and evidence positions, the study measures misleading rate (MR) and clean accuracy across fifteen systems on multiple QA datasets. Results show that prioritizing documents increases MR by 10.9–13.5 percentage points, with the highest MR when the misleading span appears at the end of the evidence.

By Shunyuan Zhou, Hao Chen, Tianyu Wang, Goose Lin, Zaiyuan Wang, Haiying Zhao
arXiv AI
Aug 25

LLM-Specific Utility for Retrieval-Augmented Generation

The paper introduces the concept of LLM‑specific utility, defining it as the performance gain a target large language model (LLM) achieves when provided with a passage compared to answering without evidence. A benchmark of utilitarian passages is built for four LLMs (Qwen3‑8B/14B/32B and Llama 3.1‑8B) across three QA datasets, revealing that each model benefits most from its own tailored evidence and that evidence optimized for other models is consistently suboptimal. The authors also create SpecUBench, a benchmark for LLM‑specific utility judgment, and show that current utility‑aware retrieval methods largely capture model‑agnostic usefulness, struggling to estimate LLM‑specific utility. "whyItMatters":"The study demonstrates that retrieval‑augmented generation must consider model‑specific evidence selection to truly improve LLM performance, highlighting a gap in existing utility‑aware methods."

By Hengran Zhang, Keping Bi, Jiafeng Guo, Jiaming Zhang, Shuaiqiang Wang, Dawei Yin, Xueqi Cheng
arXiv Computation and Language
Oct 21

The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering

arXiv:2610.26951v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) is the dominant paradigm for grounding large language models in external corpora, but its similarity-driven, chunk...

By Debopriyo Banerjee, Owais Aijaz, Ahmer Tabassum, Zhuohan Xie, Preslav Nakov, Veselin Stoyanov