The paper reports that existing knowledge‑editing benchmarks cannot evaluate the scope decision—whether a stored edit applies to a query—because they are counterfactual and lack negative examples. Using the gradient‑free editor INLAY, the authors exhaustively test every router action on 1,689 queries across three datasets and find that an oracle router achieves no gain over a static policy, and abstention never wins. The authors attribute this to the structural design of the benchmarks and demonstrate that adding a missing negative condition restores some headroom and allows abstention to win.
Every memory-based knowledge editor in the SERAC lineage depends on a scope decision: given a query, does a stored edit apply? We report that current knowledge-editing benchmarks cannot measure this d...
arXiv:2606. 14668v1 Announce Type: new Abstract: Knowledge editing systems must update selected facts while preserving nearby but irrelevant behavior.
arXiv:2609.05637v2 Announce Type: replace Abstract: A popular way to improve Retrieval-Augmented Generation (RAG) is to rewrite the user's question into several variants and search with all of them....
Clarify-Then-Search is a benchmark that tests whether large language models can ask clarification questions to improve the usefulness of deep search results. It uses 518 real-world query pairs from Baidu, where each intent query is paired with an underspecified version. The evaluation involves a clarifier asking up to three questions, a user answerer providing only explicit information, and a rewriter generating a new query that is then searched; performance is measured by a weighted nugget-recall score.
arXiv:2606. 09877v1 Announce Type: new Abstract: LLM wiki systems compile knowledge into pre-filled KV caches for efficient inference, but assume a static corpus -- an assumption that fails whenever the underlying information landscape evolves.
The paper investigates run‑to‑run variance in semantic query engines that use large language models (LLMs) to evaluate rows. By re‑executing benchmarks (SemBench, LRO‑Bench, UDA‑Bench) and logging 1.2 million model calls, the authors find that between 0.07% and 10.2% of rows change output at temperature 0, with some tasks showing up to 24% change. They show that plan amplification can cause a large fraction of answers to differ, that many benchmark comparisons are unresolved, and that voting and calibrated cascades reduce but do not eliminate variance, leading to new reporting and design guidelines.
arXiv:2607. 04281v1 Announce Type: cross Abstract: Semantic caching reduces the latency and cost of retrieval-augmented generation (RAG) by serving cached answers to semantically similar queries, but most existing methods do not model the time-varying freshness of open-web evidence.
arXiv:2608. 00106v1 Announce Type: new Abstract: Agentic systems must decide not only what answer to produce, but which reasoning and execution operations should precede it.
arXiv:2609.07559v1 Announce Type: new Abstract: How do you validate a cheap, deterministic proxy for an oracle that is expensive, rate-limited, and non-stationary? We present a protocol built on adve...
The paper investigates whether semantic entropy—a measure of disagreement among a language model’s sampled answers—can serve as a cheap signal for deciding when to route a query from a small to a larger language model. Experiments on GSM8K and other benchmarks show that semantic entropy can distinguish small‑model mistakes and improve routed accuracy, but the authors also reveal that a simple question‑difficulty rule can mimic its performance and that other factors (definition of success, benchmark design, live sampling cost) can undermine its effectiveness. They propose a checklist of checks to validate escalation signals and demonstrate how to predict when a cached‑outcome approach will fail. "whyItMatters":"The study highlights the importance of rigorous evaluation of escalation signals, showing that seemingly promising metrics can be misleading without proper controls and that practical routing decisions must account for cost and benchmark design."
The paper introduces Iris-mini and Iris-pro, two search agents trained at 35B and 397B parameter scales. They use a novel data pipeline that constructs reverse‑engineered multi‑hop queries from web hyperlinks, filters trajectories, and alternates supervised fine‑tuning with reinforcement learning in a process called SFT‑RL climbing. Evaluations on several benchmarks show that, with inference‑time context management, the agents achieve the best open‑source results in their parameter ranges.
arXiv:2608. 08853v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) routers commonly use the same scores both to select experts and to weight their already-computed outputs.