Corpus2Skill is a retrieval architecture that transforms an enterprise knowledge base into a hierarchical skill directory, enabling an LLM agent to navigate from high-level summaries to specific documents and backtrack when necessary. On an enterprise customer‑support benchmark, it outperforms single‑shot dense, hybrid, hierarchical‑retrieval, and agentic RAG baselines in answer quality and grounding, with a moderate cost tradeoff. An eleven‑dataset study shows that corpus navigation excels on single‑domain corpora with a recoverable topical taxonomy but is less effective on open‑domain factoid pools or homogeneous‑tabular corpora, providing a design guideline for knowledge‑grounded systems.
By Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh
The paper proposes a method to capture expert human judgment and reasoning for automated evaluation systems that can scale across high‑stakes domains such as healthcare, defense, finance, governance, and politics. By assembling expert networks and encoding their reasoning into structured editorial standards, the authors demonstrate that automated judges trained on these standards outperform uncalibrated frontier models on source‑quality and neutrality metrics in AI news reporting. The study validates the approach with strong inter‑annotator agreement and high F1 scores, suggesting a viable framework for expert‑calibrated evaluation in any domain where expertise is scarce.
By Robbie Goldfarb, Andrew B. Hall, Jillian Fisher, Kathryn Salam, Matt Wilde
arXiv:2610.14428v1 Announce Type: new
Abstract: Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpor...
By Istok Bako, Bonan Min, Tiffany Shao, Claire Cheng, Yashar Mehdad
arXiv:2603. 27476v2 Announce Type: replace Abstract: AI-powered people search platforms are increasingly used in recruiting, sales prospecting, and professional networking, yet no widely accepted benchmark exists for evaluating their performance.
By Wei Wang, Tianyu Shi, Shuai Zhang, Boyang Xia, Zequn Xie, Chenyu Zeng, Qi Zhang, Lynn Ai, Yaqi Yu, Kaiming Zhang, Feiyue Tang, Lei Ding
The paper introduces Evaluation-as-Search (EaS), a feedback‑driven method that adaptively probes LLM‑powered meeting assistants by focusing on natural questions likely to reveal grounding failures. Using EaS, the authors build MeetingProbe, a benchmark of over 3,000 annotated question‑answer pairs from 20 transcripts across three meeting genres and three assistants. Ablation studies show that adaptive search uncovers 2.5× more failures than random probing, revealing a capability gradient and eight recurring failure categories dominated by discourse‑pragmatic challenges.
By Sami Khairy, Yasaman Hosseinkashi, Vishak Gopal, Ross Cutler
arXiv:2608.21374v1 Announce Type: new
Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...
By Ruotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li
The paper introduces OneMillion-Bench (OMB), a benchmark comprising 400 expert-curated tasks across Law, Finance, Industry, Healthcare, and Natural Science. OMB evaluates language agents on complex, real-world scenarios that require authoritative source retrieval, conflict resolution, domain-specific rule application, and constraint decision-making. The benchmark uses a rubric-based scoring system assessing factual accuracy, logical coherence, practical feasibility, and professional compliance to differentiate agent performance on expert-level problems.
By Yang Liu, Jiaqi Li, Jun Bai, Qianyu Yang, Xiaobo Hu, Tao Peng, Zaiyuan Wang, Ran Tian, Jiayun Dong, Chun Zhang, Zixia Jia, Kaiyuan Chen, Yixin Ren, Yang Liu, Yanglihong Xiao, Lingyue Yin, Tiliang Duan, Ge Zhang, Gang Yao, Hao Chen, Yuan Gong, Jianpeng Jiao, Zilong Zheng
arXiv:2609.21841v1 Announce Type: new
Abstract: Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically val...
By Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona Zahid
SEEK (Skill‑Routed Evaluation with Evolvable Knowledge) is a framework that externalizes search evaluation criteria into a skill bank, dynamically routes relevant skills for each query‑result pair, and uses a task‑adapted listwise evaluator to generate page‑level judgments and failure‑mode attribution. It employs a two‑stage training pipeline to align evaluation with human preferences and a replay‑gated skill bank to incorporate new evaluation knowledge without retraining the model. Experiments on industrial short‑video search demonstrate that SEEK improves listwise quality evaluation accuracy and significantly advances attribution diagnosis, leading to its deployment at Kuaishou with over 400 million daily active users.
By Zhongxin Huang, Songyang Li, Renzhe Zhou, Feiran Zhu, Chenglei Dai, Zhen Xiao, Xuanping Li, Jingwei Zhuo
CARGO is a framework for evaluating agentic AI systems in production that addresses the problem of reference-instance divergence (RID), where reference-based judges penalize correct answers that involve different entity identifiers. It treats retrieved references as procedural exemplars, grounds judgments in the live instance’s context, assigns a three-way status to claims, and gates evaluation by retrieval confidence. Using the CARGO-Bench diagnostic suite, CARGO eliminates false penalties and improves discrimination while revealing a limitation in detecting procedural corruptions.
By Mukul Chhabra, Shail Patel, Luigi Medrano
SEEK (Skill‑Routed Evaluation with Evolvable Knowledge) is a framework that externalizes search evaluation criteria into a skill bank, dynamically routes relevant skills for each query‑result pair, and uses a task‑adapted listwise evaluator to generate page‑level judgments and failure mode attribution. It employs a two‑stage training pipeline to align the evaluator with human preferences and a replay‑gated skill bank to incorporate new evaluation knowledge without retraining the model. Experiments on Kuaishou’s short‑video search demonstrate that SEEK improves listwise quality evaluation accuracy and enhances attribution diagnosis, leading to better online search evaluation at scale.
Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval introduces CoverageBench-Negate, a new benchmark that augments the existing CoverageBench with controlled contradictions. For each evidence nugget, a negated counterpart is created and its supporting documents are edited to support the negation while preserving context, resulting in about 63,000 nugget‑document pairs across six datasets. The benchmark includes rich construction metadata, LLM judgments, and a human audit, and baseline experiments show BM25 can distinguish original from edited evidence better than dense retrievers.
By Pasha Abrishamchian, Eugene Yang, Andrew Yates, Nazli Goharian, Ophir Frieder