← Back to all news
arXiv AI By Aditya Pratap Singh

On Scope Classification and Current Knowledge-Editing Benchmarks: A Negative Result, with INLAY as a Gradient-Free Case Study

Read the original on arXiv AI →

The paper reports that existing knowledge‑editing benchmarks cannot evaluate the scope decision—whether a stored edit applies to a query—because they are counterfactual and lack negative examples. Using the gradient‑free editor INLAY, the authors exhaustively test every router action on 1,689 queries across three datasets and find that an oracle router achieves no gain over a static policy, and abstention never wins. The authors attribute this to the structural design of the benchmarks and demonstrate that adding a missing negative condition restores some headroom and allows abstention to win.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 24

Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration

Clarify-Then-Search is a benchmark that tests whether large language models can ask clarification questions to improve the usefulness of deep search results. It uses 518 real-world query pairs from Baidu, where each intent query is paired with an underspecified version. The evaluation involves a clarifier asking up to three questions, a user answerer providing only explicit information, and a rewriter generating a new query that is then searched; performance is measured by a weighted nugget-recall score.

By Deqiang Huang, Jingbo Zhou, Xinjiang Lu, Tong Xu, Hua Wu, Enhong Chen
llmsbenchmarks
More like this →
arXiv AI
Oct 12

Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

The paper investigates run‑to‑run variance in semantic query engines that use large language models (LLMs) to evaluate rows. By re‑executing benchmarks (SemBench, LRO‑Bench, UDA‑Bench) and logging 1.2 million model calls, the authors find that between 0.07% and 10.2% of rows change output at temperature 0, with some tasks showing up to 24% change. They show that plan amplification can cause a large fraction of answers to differ, that many benchmark comparisons are unresolved, and that voting and calibrated cascades reduce but do not eliminate variance, leading to new reporting and design guidelines.

By Oktie Hassanzadeh, Dharmashankar Subramanian
llmsbenchmarks
More like this →