arXiv Machine Learning By Heejin Choi

Beyond Endpoint Scores: Time- and Capacity-Conditioned Evaluation of Continual Knowledge Updating

Read the original on arXiv Machine Learning →

The paper argues that evaluating continual knowledge‑updating methods solely at a final checkpoint and a single adapter rank can be misleading. By fixing a periodic hierarchy and comparing it to cumulative replay on a 24‑month Wikidata stream, the authors show that the apparent best method changes depending on the evaluation month, replay LoRA rank, and query formulation. They recommend reporting performance trajectories and capacity sweeps, and only declaring a robust winner when the ranking remains stable across the evaluation region.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
2d ago

LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture

The paper introduces LSREP, a Longitudinal State‑Replay Evaluation Protocol designed to assess how conversational memory evolves over time, incorporating ordered replay, lifecycle schedules, repeated probes, evolving reference answers, and mechanism‑fidelity checks. It applies LSREP to ICE v2, a local‑first memory middleware, and reports that on three ordinary‑density datasets ICE v2 achieves near‑zero mean quality difference from vector‑RAG while using fewer fragments but slightly more prompt tokens, yet fails catastrophically on a dense dataset. In a public diagnostic, ICE v2 underperforms pure vector‑RAG on LongMemEval, revealing significant multi‑session and temporal failures and a quality‑cost trade‑off rather than superior efficiency.

By Deepesh Sonar
Hugging Face Trending Papers
Sep 3

What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation

The paper investigates how temporal aggregation of token scores during decoding-time KV cache compression affects eviction decisions. It shows that using an exponential‑moving‑average (EMA) aggregation keeps ranking changes minimal for many scorer variants, while others like KeyDiff or recency significantly alter rankings and degrade performance. Building on this, the authors propose InertiaKV and its lazy variant, which achieve 1.34–1.46× faster decoding than full refresh, and also evaluate a score‑free approach that slightly improves quality while eliminating further scoring.

arXiv AI
Sep 4

What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation

The paper investigates how temporal aggregation and ranking preservation affect aggressive decoding-time KV cache eviction. It shows that exponential‑moving‑average (EMA) aggregation makes many scorer modifications indistinguishable at the eviction‑set level, while certain variants (KeyDiff, key norm, recency, learned scorer) alter rankings and degrade performance. Building on this, the authors propose InertiaKV and its lazy variant, achieving 1.34–1.46× faster decoding, and also explore a score‑free decoding approach that slightly improves quality while eliminating further scoring.

By Bo Zeng, Yu Zhao, Yefeng Liu, Zhihong Lu, Xuanfan Ni, Xintong Wang
arXiv AI
Aug 26

Matched Excess-Outranker Regularization for Candidate-Set Interference in Continual Knowledge Graph Embedding

The paper introduces Matched Excess-Outranker Regularization (MEOR), a new host-level objective for continual knowledge graph embedding that addresses candidate-set interference caused by entity admission. MEOR compares newcomer pressure with matched old references, applying a one-sided penalty only when newcomers outcompete these references, thereby preserving the learner’s signal for legitimate new entities. Experiments on ENTITY-ComplEx and FBInc datasets show that MEOR improves historical current-universe mean reciprocal rank and reduces candidate-set interference, outperforming several baseline regularizers.

By Hao Ren, Junbin Gao, Jiaojiao Jiang