arXiv AI
CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Aug 27
Corpus2Skill: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG
Corpus2Skill is a retrieval architecture that transforms an enterprise knowledge base into a hierarchical skill directory, enabling an LLM agent to navigate from high-level summaries to specific documents and backtrack when necessary. On an enterprise customer‑support benchmark, it outperforms single‑shot dense, hybrid, hierarchical‑retrieval, and agentic RAG baselines in answer quality and grounding, with a moderate cost tradeoff. An eleven‑dataset study shows that corpus navigation excels on single‑domain corpora with a recoverable topical taxonomy but is less effective on open‑domain factoid pools or homogeneous‑tabular corpora, providing a design guideline for knowledge‑grounded systems.
arXiv AI
3d ago
Distilling Expert Judgment at Scale
The paper proposes a method to capture expert human judgment and reasoning for automated evaluation systems that can scale across high‑stakes domains such as healthcare, defense, finance, governance, and politics. By assembling expert networks and encoding their reasoning into structured editorial standards, the authors demonstrate that automated judges trained on these standards outperform uncalibrated frontier models on source‑quality and neutrality metrics in AI news reporting. The study validates the approach with strong inter‑annotator agreement and high F1 scores, suggesting a viable framework for expert‑calibrated evaluation in any domain where expertise is scarce.
arXiv AI
Oct 15
Frustratingly Easy Agentic RAG via Portable Skills: A Case Study in Customer Support
arXiv:2610.14428v1 Announce Type: new Abstract: Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpor...
arXiv AI
Jul 28
PeopleSearchBench: A Multi-Dimensional Benchmark for Evaluating AI-Powered People Search Platforms
arXiv:2603. 27476v2 Announce Type: replace Abstract: AI-powered people search platforms are increasingly used in recruiting, sales prospecting, and professional networking, yet no widely accepted benchmark exists for evaluating their performance.
arXiv AI
Aug 24
Evaluation-as-Search: Adaptive Discovery of Grounding Failures in Meeting Assistants
The paper introduces Evaluation-as-Search (EaS), a feedback‑driven method that adaptively probes LLM‑powered meeting assistants by focusing on natural questions likely to reveal grounding failures. Using EaS, the authors build MeetingProbe, a benchmark of over 3,000 annotated question‑answer pairs from 20 transcripts across three meeting genres and three assistants. Ablation studies show that adaptive search uncovers 2.5× more failures than random probing, revealing a capability gradient and eight recurring failure categories dominated by discourse‑pragmatic challenges.
arXiv AI
Aug 25
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
arXiv:2608.21374v1 Announce Type: new Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...