← Back to all news
arXiv AI

Completed Pairs Hide Capped Failures: A ReVerPi Case Study of Selective Context Projection

The study examines how context projection—replacing older tool observations with concise, addressable excerpts—affects performance in ReVerPi, a Pi extension that archives observations and matches full to projected continuations. Across 86 source‑reading runs and 641 model requests, 15 completed pairs achieved identical success rates (12/15 per arm), while 12 boundary runs halted when the first arm failed, revealing that projection can reduce logical tokens by 25% but increase median pair tokens by 29% and total suffix requests from 35 to 55. The analysis highlights the importance of retaining all intervention boundaries, executing both arms independently, and reporting completion, token usage, and interaction details to accurately assess stopping rules and resource aggregation.

Related stories

arXiv AI
Sep 4

It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories

The paper investigates whether large language models’ reasoning traces truly contain early, informative signals or merely reflect budget and difficulty confounds. Using a restart‑controlled truncation probe, the authors compare continuation success rates against from‑scratch restart curves across 178 problem‑model pairs, finding that only one case shows prefix‑limited success and that continuing a model’s own prefix generally outperforms restarting. A difficulty‑controlled test and two generation‑free analyses reveal that early internal signals do not carry outcome information beyond a problem‑difficulty baseline, underscoring the need for proper counterfactual controls.

By Yigit Utku Bulut
llms
More like this →
arXiv AI
Sep 7

Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines

The paper investigates whether the object selected in a grounded language‑model pipeline actually reaches the reader, a failure that can break the handoff between stages. By auditing 600 HybridQA questions across three selector families, the authors find that exact key lookup and title matching recover the selected object in all 1,463 resolvable records, but body‑only BM25 omits it in 26.6% of cases at cutoff five, while hybrid retrieval with reranking omits it only 1.0%. The study also shows that misalignment between selected and retrieved objects can reduce exact match scores by up to 31 points, and introduces the Returned‑Object Profile (ROP) as a tool for reproducible auditing.

By Siddharth Vohra, Runmin Jiang, Xiaomo Li, Min Xu
ragbenchmarks
More like this →
arXiv AI
Sep 16

LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture

The paper introduces LSREP, a Longitudinal State‑Replay Evaluation Protocol designed to assess how conversational memory evolves over time, incorporating ordered replay, lifecycle schedules, repeated probes, evolving reference answers, and mechanism‑fidelity checks. It applies LSREP to ICE v2, a local‑first memory middleware, and reports that on three ordinary‑density datasets ICE v2 achieves near‑zero mean quality difference from vector‑RAG while using fewer fragments but slightly more prompt tokens, yet fails catastrophically on a dense dataset. In a public diagnostic, ICE v2 underperforms pure vector‑RAG on LongMemEval, revealing significant multi‑session and temporal failures and a quality‑cost trade‑off rather than superior efficiency.

By Deepesh Sonar
ragnlp
More like this →
arXiv AI
Aug 25

XTC: Head-Aware Sampling by Excluding Top Choices

XTC (Exclude Top Choices) is a lightweight, head‑aware decoding operator that improves diversity in autoregressive language models by removing overly probable tokens that dominate the next‑token distribution. It works by identifying tokens above a plausibility threshold, probabilistically excluding the dominant choices, and renormalizing the remaining distribution. Across 60 experiments on models such as Gemma 3 and DeepSeek R1, XTC boosts Distinct‑2 scores by 11–15 % and cuts repeat trigrams by 27–47 %, while a Mechanical Turk study shows a 62.3 % preference for XTC‑generated text without loss of fluency.

By Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder, Sanjay Basu, Ravid Shwartz-Ziv
llmsefficiency
More like this →
arXiv AI
Jun 6

Answer Presence Drives RAG Rewriting Gains

arXiv:2606. 05633v1 Announce Type: new Abstract: Retrieval-augmented QA pipelines often route retrieved passages through an LLM \emph{rewriter} before a smaller reader, lifting F1 by tens of points on multi-hop benchmarks; this gain is typically credited to improved evidence quality.

By Yuejie Li, Yueying Hua, Ke Yang, Li Zhang, Yueping He, Yueping He, Ruiqi Li, Bolin Chen, Tao Wang, Bowen Li, Chengjun Mao
llmsragbenchmarks
More like this →