arXiv Machine Learning
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 30
Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation
arXiv:2606. 29947v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes.
arXiv Computation and Language
Sep 3
Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection
The paper proposes an incremental pooled LLM evaluation method for selecting retrieval models in production Retrieval-Augmented Generation (RAG) systems. By having a language model judge the union of documents retrieved by current candidates and expanding the pool only with new documents from added systems, the approach reuses judgments across all systems. Experiments on four benchmarks and a financial news QA deployment show strong correlation with gold-standard rankings, high preservation of pairwise orderings, and significant cost savings—up to 4.9× lower evaluation cost and 65–80% judgment reuse.
arXiv Machine Learning
Aug 12
When Do Anchor-Based Pointwise LLM Rerankers Help? Retriever Quality, Statistical Scope, and Anchor Design
arXiv:2608. 10528v1 Announce Type: cross Abstract: Anchor-based pointwise LLM reranking scores each candidate against a shared reference passage to recover cross-document context at pointwise cost.
arXiv Computation and Language
Oct 23
Beyond Recall Saturation: Scaling Hard-Negative Reranker Training for Industrial Legal Retrieval
The paper investigates how scaling hard‑negative training for rerankers can improve industrial legal retrieval systems once embedding recall saturates. On Chinese legal benchmarks, a pipeline that first uses embedding‑mined negatives and then self‑mined negatives yields the best trade‑off, and a 4B‑parameter Stage2 reranker outperforms the native 4B model on STARD, CSAID, and internal datasets. The study demonstrates that evaluating retrieval and reranking stages separately is beneficial for optimizing industrial retrieval performance.
arXiv Computation and Language
Aug 25
Robustness of IR Models to Collection Growth
The paper investigates how adding non‑relevant documents to a collection affects the performance of Information Retrieval (IR) models. It distinguishes between Multi‑Document‑Agnostic (MDA) models, which do not rely on other documents, and Multi‑Document‑Dependent (MDD) models, which do, and evaluates both on two collections with little topic overlap. Results show that all models suffer some performance loss when non‑relevant documents are added, but MDA models outperform MDD models in retrieval tasks, while both perform similarly in reranking.
arXiv Machine Learning
Oct 28
Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation
The paper investigates whether System‑1‑style decision models can replace dedicated rerankers in retrieval‑augmented generation (RAG) systems. After screening 21 open‑weight models on 11 benchmarks, the authors compare a promising decision model with a dedicated reranker on identical candidate lists. While limiting candidate text length improves reliability and makes some configurations competitive, a tuned reranker remains faster and uses less GPU memory, indicating that replacing a reranker requires balancing quality, latency, and memory.