← Back to all news
arXiv Machine Learning By Andre Bacellar

Predictable Failure in Multi-Hop Retrieval: Score-Distributional Confidence Scoring and Abstention

Read the original on arXiv Machine Learning →

The paper shows that multi‑hop retrieval failures cluster in predictable subpopulations and formalizes this with two theoretical results: (1) confident‑failure reduction is possible only when retrieval features carry mutual information about success, and (2) no single ANN score feature dominates across all failure regimes. Building on these insights, the authors introduce RegimeAbstain, which computes a Retrieval Confidence Score (RCS) from up to nine query‑ANN structural features and uses it to calibrate an abstention policy. Across three benchmarks and two retrieval architectures, RCS achieves the best or co‑best AUC‑AC and significantly reduces the Confident‑Wrong‑Answer Rate, demonstrating its effectiveness and domain‑agnostic applicability.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Aug 25

DynaKRAG: A Unified Framework for Learnable Evidence Control in Multi-Hop Retrieval-Augmented Generation

DynaKRAG is a unified framework that learns a state‑conditioned policy to control evidence acquisition in multi‑hop retrieval‑augmented generation. It uses a deterministic validity layer to build an action set, a learned continuation gate to decide between generating an answer or gathering more evidence, and an advantage scorer to rank evidence operations by predicted gain. Across HotpotQA, 2Wiki, and MuSiQue with various backbone models, DynaKRAG achieves top EM and F1 scores, improves token and retrieval efficiency, and enables terminal evidence compression that reduces context size while boosting answer quality.

By Chenyu Zhou, Yaqi Wu, Xiaolei Guo, Jiaqi Huang, Xianfa Zhang, Junxu Zhang, Zhuo Yu, Zhubo Shi, Jianghao Lin, Dongdong Ge
llmsrag
More like this →
arXiv AI
Aug 28

Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay

The paper introduces matched trajectory replay, a protocol that fixes answer states, evidence points, budgets, and action costs to evaluate how confidence signals influence agent actions. Using this method, the authors compare raw verbalized confidence with post‑hoc isotonic calibration across six model‑dataset pairs, finding that calibration can significantly improve accuracy of committed answers but may reduce coverage and increase retrieval usage. The study concludes that calibration helps interpret commitment risk but does not predict the benefit of additional retrieval, indicating the need for separate value‑of‑information estimates.

By Prateek Chhikara
llmsagents
More like this →