← Back to all news
arXiv Computation and Language By Xuan Yao, Li Shuping, Dai Yang, Zhou Yi, Ke-Wei Huang

DelistBench: Evaluating Search-Enabled LLMs for Auditable Corporate-Event Database Completion

Read the original on arXiv Computation and Language →

The paper introduces Search-to-Record, a database‑assurance task where search‑enabled large language models reconstruct corporate‑event records from public sources, and DelistBench, a 1,200‑record benchmark for security‑level delisting announcements. Five models were evaluated in both closed‑book and web‑enabled settings, showing that web access improves announcement‑date accuracy by 34.0–48.0 percentage points and event‑status accuracy by 2.8–21.7 points, with the best system achieving 81.5% overall joint accuracy within seven days. The study also demonstrates that low‑cost web systems can approach the best system’s accuracy and that risk‑based triage can identify low‑error subsets, though a significant portion of cases still requires review.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Jun 12

Fin-RATE: A Real-world Financial Analytics and Tracking Evaluation Benchmark for LLMs on SEC Filings

arXiv:2602. 07294v4 Announce Type: replace-cross Abstract: With the increasing deployment of Large Language Models (LLMs) in the finance domain, LLMs are increasingly expected to parse complex regulatory disclosures.

By Yidong Jiang, Junrong Chen, Eftychia Makri, Jialin Chen, Peiwen Li, Ali Maatouk, Leandros Tassiulas, Eliot Brenner, Bing Xiang, Rex Ying
llmsragbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 3

Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection

The paper proposes an incremental pooled LLM evaluation method for selecting retrieval models in production Retrieval-Augmented Generation (RAG) systems. By having a language model judge the union of documents retrieved by current candidates and expanding the pool only with new documents from added systems, the approach reuses judgments across all systems. Experiments on four benchmarks and a financial news QA deployment show strong correlation with gold-standard rankings, high preservation of pairwise orderings, and significant cost savings—up to 4.9× lower evaluation cost and 65–80% judgment reuse.

By Max Nelson, Hanoz Bhathena, Aviral Joshi, Saket Sharma
llmsragbenchmarks
More like this →
arXiv AI
Aug 7

SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

arXiv:2608. 05212v1 Announce Type: new Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.

By Zhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao
llmsagentsbenchmarks
More like this →
arXiv AI
Sep 1

Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs

The paper evaluates how modern large language models use internal web search to answer factual questions. Using 783 static queries and 288 dynamic queries, the authors find that enabling retrieval improves accuracy on static questions but hurts confidence calibration. On dynamic queries, models often retrieve but still achieve less than 70% accuracy, mainly due to poor query formulation and source selection, indicating that internal web search works better as a quick verification tool than a full information‑retrieval system.

By Sahil Kale
llms
More like this →