The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper investigates how conversational Retrieval-Augmented Generation (RAG) benchmarks that rely on relevance judgments to determine context sufficiency can be misleading. By re‑running official baselines on the RETECO track and evaluating with both large‑language‑model judges and a blind audit, the authors show that stronger retrievers produce a higher proportion of contexts lacking gold documents yet still containing the answer, leading to a sharp drop in shortcut precision from 0.68 to 0.10. The study demonstrates that many false misses arise from hard negatives, gold content from other turns, or neighboring chunks, and that using retrieval scores alone as a sufficiency detector can underestimate answerability by up to 0.19 in AUROC.
The paper investigates whether the object selected in a grounded language‑model pipeline actually reaches the reader, a failure that can break the handoff between stages. By auditing 600 HybridQA questions across three selector families, the authors find that exact key lookup and title matching recover the selected object in all 1,463 resolvable records, but body‑only BM25 omits it in 26.6% of cases at cutoff five, while hybrid retrieval with reranking omits it only 1.0%. The study also shows that misalignment between selected and retrieved objects can reduce exact match scores by up to 31 points, and introduces the Returned‑Object Profile (ROP) as a tool for reproducible auditing.
TRACE is a system designed to bridge the grounding contract gap in LitTraceQA by combining target-aware retrieval, independent typed evidence localization, multimodal table extraction, and schema-driven table construction. It indexes 27,487 papers using multiple representations while preserving question targets, predicts observation units for tables, and assembles rows with evaluator-compatible key normalization. On the official test set, TRACE achieves a 0.760613 overall score, with high paper F1, evidence F1, and multiple-choice accuracy, though table-row and macro cell performance remain lower.
AtomCite is an agentic framework that verifies and corrects page‑level citations in multi‑page documents by parsing answers into claims, checking each claim against the cited page image, and applying a deterministic repair policy. The authors introduce DocCite, the first benchmark for this task, built on MP‑DocVQA and DUDE, containing 928 injected instances and 1,909 verified natural errors. Across Gemini, Claude, and GPT models, AtomCite achieves about 93% verification accuracy and improves citation precision from 34% to 87‑90%, while also enhancing hallucination detection in open‑source models.
The paper investigates why large‑language‑model coding agents rarely request a second chunk of tool output, focusing on the precision‑at‑1 rate ($p_1$) of the gold item appearing first in the first chunk. In a benchmark of 500 software‑engineering tasks, the authors compare six value functions and find that increasing $p_1$ does not systematically improve downstream accuracy; the agent can recover the correct answer from any position within the chunk. Adding file‑metadata signals to a keyword scorer actually reduces $p_1$, while a parameter‑free keyword scorer improves $p_1$ but still fails to boost overall accuracy.
Re:CAP is a reference‑free audit loop for retrieval‑augmented generation (RAG) pipelines that probes for missing documents instead of enumerating all relevant ones. It identifies covered topics, generates probing questions, retrieves candidate documents, and uses an LLM judge to keep only those that add new information. On several benchmarks, Re:CAP recovers a significant portion of gold documents that flat BM25 or hybrid retrieval misses, and human evaluation shows most of these documents add new information.