arXiv AI
PUFFER: Incremental Fuzzy Deduplication for Continuously Evolving Corpora
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 24
RAG Deserves an Index: Why Ingest-Time Compilation Beats Query-Time Interpretation
The paper argues that retrieval‑augmented question‑answering systems should perform semantic compilation at ingest time rather than re‑deriving meaning at query time. By building a maintained structure—incrementally updated embeddings and validated atomic claims—read operations become far cheaper, with experimental results showing higher accuracy and lower token usage compared to traditional chunk‑based retrieval. The authors present two proofs: cheaper incremental updates and superior performance on broadcast‑interview transcripts, suggesting a new systems agenda for compilation and read planning.
Hugging Face Trending Papers
Aug 4
Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining
Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challenging.
arXiv AI
Jun 2
Don't Ask the LLM to Track Freshness: A Deterministic Recipe for Memory Conflict Resolution
arXiv:2606. 01435v1 Announce Type: new Abstract: LLM-based memory systems increasingly maintain facts that evolve over time, where a recurring failure is conflict resolution: when a fact has multiple contradictory values, which should the agent return?
arXiv AI
Sep 25
Ingest-Time Fact Compilation for Cost-Efficient and Reliable Question Answering over Revised Corpora
The paper introduces ingest‑time fact compilation, an architecture that preprocesses and compiles corpus data into self‑contained facts with resolved revisions, deletions, and source trust. By storing this compiled state, query‑time models can retrieve answers directly, avoiding costly reconstruction from raw passages. Experiments show that this approach reduces read cost per question by 12.89× and token usage by 21.6× while maintaining accuracy.
arXiv AI
Sep 17
GVD: Governed Versioning and Deduplication for Document Repositories
GVD (Governed Versioning and Deduplication) is a framework that unifies cross‑document version linking with rule‑level conflict resolution under an auditable update policy. It assigns incoming documents to version families via bidirectional rule alignment, then compares their rules against family memory to detect duplicates, contradictions, asymmetric refinements, and new knowledge, using Counterfactual Span Probing (CSP) to correct neutral misclassifications. In a test on 120 enterprise documents across 59 version families, GVD achieved an F1 of 0.97 for version‑family construction and 0.94 for rule‑level consistency, with CSP improving rule consistency from 0.90 to 0.94.
arXiv AI
Sep 11
Fortunate Recall: Ontology-Driven Memory Lifecycle Management for Persistent Coherence in LLMs
Fortunate Recall (FR) introduces an ontology-driven policy layer that categorizes personal facts into over ten behavioral types and applies tailored lifecycle rules—such as differential decay, supersession, and event-time validity—to manage memory persistence in large language models. The FR-Bank implementation, independent of underlying infrastructure, achieves a 76.9% pass rate on the new LifecycleBench benchmark and improves LongMemEval-S performance, while significantly reducing confabulation rates compared to prior systems. Ablation studies show that the generic lifecycle metadata drives correctness, whereas the behavioral ontology enhances calibration and reduces downstream hallucinations.