arXiv AI
What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 9
Decoy-Calibrated Failure Audits for Language Models
arXiv:2606. 09046v1 Announce Type: new Abstract: Useful audits reveal not only how often a model fails, but also where its failures concentrate.
arXiv Computation and Language
Sep 10
When Auditors Fabricate: Batch-Size Degradation and Confident Hallucination in LLM Detection of Planted Document Contamination
arXiv:2609.09696v1 Announce Type: new Abstract: Large language models are increasingly proposed as automated auditors of document quality, yet their reliability as detectors of planted errors is poor...
arXiv Computation and Language
Sep 10
In RAG We Trust? Measuring Robustness of Retrieval-Augmented Generation Under Document Poisoning
arXiv:2609.09243v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) grounds a language model in retrieved documents, which reduces hallucination but creates a new attack surface: i...
arXiv Computation and Language
Oct 15
Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States
arXiv:2607.11414v2 Announce Type: replace Abstract: Large language models (LLMs) in financial applications fail most consequentially when they are confidently wrong. Hedged, uncertain answers invite...
arXiv Computation and Language
Oct 9
CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
CiteVQA is a benchmark that evaluates multimodal large language models on both answering document questions and providing element‑level bounding‑box citations for the evidence used. It contains 1,897 questions across 711 PDFs in seven domains and two languages, with ground‑truth citations generated via an automated pipeline and strict quality control. The evaluation metric, Strict Attributed Accuracy (SAA), rewards a prediction only when both the answer and the cited region are correct, revealing widespread attribution hallucination in current models.
arXiv Computation and Language
Sep 22
Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines
Re:CAP is a reference‑free audit loop for retrieval‑augmented generation (RAG) pipelines that probes for missing documents instead of enumerating all relevant ones. It identifies covered topics, generates probing questions, retrieves candidate documents, and uses an LLM judge to keep only those that add new information. On several benchmarks, Re:CAP recovers a significant portion of gold documents that flat BM25 or hybrid retrieval misses, and human evaluation shows most of these documents add new information.