arXiv:2609.15660v1 Announce Type: cross
Abstract: Retrieval-augmented generation (RAG) grounds a language model's answers on retrieved external knowledge and returns each answer with citations that i...
By Guo Fuzheng
The paper introduces REASONS, a benchmark of 12,723 sentence-level citation instances across 12 arXiv subject categories, to evaluate scientific citation attribution under different evidence conditions. It proposes a dual-metric framework—Abstention Rate (AR) and Hallucination Rate (HR)—to balance reliability and responsiveness. Experiments with proprietary and open-source LLMs across various prompting and retrieval settings show that advanced Retrieval-Augmented Generation (RAG) reduces hallucinations but increases abstention, while adversarial metadata can push hallucination rates above 85%. Human evaluation confirms a high ratio of factual hallucinations to acceptable paraphrases, underscoring the need for systems that can appropriately abstain under uncertainty.
By Deepa Tilwani, Yash Saxena, Seyedali Mohammadi, Ankur Padia, Edward Raff, Amit Sheth, Srinivasan Parthasarathy, Manas Gaur
The paper introduces REASONS, a benchmark comprising 12,723 sentence-level citation instances across 12 arXiv subject categories, to evaluate scientific citation attribution by large language models. It proposes a dual-metric framework—Abstention Rate (AR) and Hallucination Rate (HR)—to assess the trade-off between reliability and responsiveness. Experiments on proprietary and open-source LLMs under various prompting and retrieval settings show that advanced Retrieval-Augmented Generation (RAG) reduces hallucinations but may increase abstention, while retrieval-augmented variants often maintain near-zero abstention. Human evaluation reveals a high ratio of factual hallucinations to acceptable paraphrases, underscoring the need for systems that can appropriately abstain under uncertainty.
By Deepa Tilwani, Yash Saxena, Seyedali Mohammadi, Ankur Padia, Edward Raff, Amit Sheth, Srinivasan Parthasarathy, Manas Gaur
The paper investigates how the granularity of citations—sentence, paragraph, or document level—affects the performance of large language models in attributed generation tasks. Across models ranging from 8B to 120B parameters, enforcing fine‑grained, sentence‑level citations consistently reduces performance, with median losses of 40% and up to 338% on specific tasks, while overall answer correctness remains largely unchanged. The study finds that attribution quality peaks at intermediate, paragraph‑level granularity, suggesting that overly fine citations break semantic dependencies and overly coarse ones add noise, and that the optimal granularity depends on model scale and the amount of evidence required.
whyItMatters":"The findings reveal that the conventional preference for fine‑grained citations can actually harm model performance, indicating that attribution standards should be tailored to the model’s semantic scope rather than fixed by convention."
By Hexuan Wang, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi
arXiv:2608.21376v1 Announce Type: cross
Abstract: Many NLP tasks require systems to provide attribution in their outputs--i.e. citations to grounding sources. Attribution serves as a bulwark against...
By Yu Hou, Hal Daum\'e III, Rachel Rudinger, William Walden
The paper introduces a method for identifying the dominant knowledge source behind large language model (LLM) outputs, distinguishing between faithfulness violations (misuse of provided context) and factuality violations (errors in internal knowledge). A simple linear probe trained on hidden representations can reliably classify this source, and the authors present AttriWiki, a self‑supervised pipeline that generates labeled training data by prompting models to recall withheld entities or read them from context. Probes trained on AttriWiki achieve high Macro‑F1 scores across several models and datasets, generalize zero‑shot to a benchmark, and show that attribution mismatches can increase error rates by up to 70%.
"whyItMatters":"The study demonstrates that knowing the source of an LLM’s answer is crucial for effective mitigation of hallucinations, as attribution mismatches significantly raise error rates."
By Ivo Brink, Alexander Boer, Dennis Ulmer