arXiv Machine Learning
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 30
Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation
arXiv:2606. 29947v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes.
arXiv Computation and Language
Sep 3
Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection
The paper proposes an incremental pooled LLM evaluation method for selecting retrieval models in production Retrieval-Augmented Generation (RAG) systems. By having a language model judge the union of documents retrieved by current candidates and expanding the pool only with new documents from added systems, the approach reuses judgments across all systems. Experiments on four benchmarks and a financial news QA deployment show strong correlation with gold-standard rankings, high preservation of pairwise orderings, and significant cost savings—up to 4.9× lower evaluation cost and 65–80% judgment reuse.
arXiv Machine Learning
Aug 12
When Do Anchor-Based Pointwise LLM Rerankers Help? Retriever Quality, Statistical Scope, and Anchor Design
arXiv:2608. 10528v1 Announce Type: cross Abstract: Anchor-based pointwise LLM reranking scores each candidate against a shared reference passage to recover cross-document context at pointwise cost.
arXiv Computation and Language
Aug 25
Robustness of IR Models to Collection Growth
The paper investigates how adding non‑relevant documents to a collection affects the performance of Information Retrieval (IR) models. It distinguishes between Multi‑Document‑Agnostic (MDA) models, which do not rely on other documents, and Multi‑Document‑Dependent (MDD) models, which do, and evaluates both on two collections with little topic overlap. Results show that all models suffer some performance loss when non‑relevant documents are added, but MDA models outperform MDD models in retrieval tasks, while both perform similarly in reranking.
arXiv AI
Jun 30
Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present?
arXiv:2606. 28367v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is routinely extended with methods meant to improve retrieval: query expansion, hierarchical and cross-document summarization, graph-based expansion, per-query routing, rank fusion, and corrective re-retrieval.
arXiv AI
Sep 2
Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation
The study evaluates large language models (LLMs) as rerankers in conversational movie recommendation, comparing proprietary, open-weight, and fine-tuned LLMs against collaborative-filtering and sequential baselines on the ReDial benchmark. Results show that the best proprietary LLM achieves an NDCG@10 of 0.1497 with a shared semantic candidate pool, outperforming non-LLM baselines, while open-weight LLMs do not surpass a tuned shallow autoencoder under the same protocol. The analysis also highlights that reranker performance is highly sensitive to candidate generation, pool size, scoring policy, and decoding temperature, suggesting these factors should be reported as standard evaluation fields.