Towards Data Science

Designing a Persistent Knowledge Layer That Refuses to Guess

RAG Retrieves, It Never Remembers. A vendor-neutral blueprint for applications that accumulate understanding.

arXiv Machine Learning
Sep 15

Realistic Continual Learning Approach using Pre-trained Models

arXiv:2404.07729v2 Announce Type: replace Abstract: Continual learning (CL) evaluates adaptability in learning solutions to retain knowledge. Our research addresses the challenge of catastrophic forg...

By Nadia Nasri, Carlos Guti\'errez-\'Alvarez, Sergio Lafuente-Arroyo, Saturnino Maldonado-Basc\'on, Roberto J. L\'opez-Sastre
arXiv AI
Sep 15

Homeostatic Continual Learning

The paper introduces a new approach called Homeostatic Continual Learning, designed to allow an AI agent to learn continuously in a changing environment without catastrophic forgetting. The method identifies outliers in environmental data when the agent’s output deviates, enabling the agent to incrementally refine its model and policy across increasingly diverse contexts. The authors also propose extending the method to build a world model that factorizes objects into features, abstracts them into comparable concept instances, and maps concepts to intents via features, while outlining necessary future work and broader AI connections.

By Yue Jin
arXiv AI
Sep 15

K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments

K-Bench is a new benchmark designed to evaluate large language model (LLM) unlearning when the models are deployed as agents. Unlike previous benchmarks that only inspect the final answer, K-Bench examines all six channels of a ReAct agent—including chain-of-thought, tool calls, tool observations, and elicited summaries—to determine if a secret is leaked. The benchmark measures leakage for secrets placed in the model weights, prompt, or retrieval store, and finds that many existing unlearning methods fail to prevent leaks in deployed agents, especially when secrets reside in the prompt or retrieval store.

By Guangsheng Yu, Yanna Jiang, Qin Wang, Baihe Ma, Xu Wang
arXiv AI
Jun 6

Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

arXiv:2606. 05661v1 Announce Type: new Abstract: Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it.

By Parth Asawa, Christopher M. Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, Asim Biswal, Vincent Sunn Chen, Frederic Sala, Matei Zaharia, Joseph E. Gonzalez
arXiv AI
Sep 11

Fortunate Recall: Ontology-Driven Memory Lifecycle Management for Persistent Coherence in LLMs

Fortunate Recall (FR) introduces an ontology-driven policy layer that categorizes personal facts into over ten behavioral types and applies tailored lifecycle rules—such as differential decay, supersession, and event-time validity—to manage memory persistence in large language models. The FR-Bank implementation, independent of underlying infrastructure, achieves a 76.9% pass rate on the new LifecycleBench benchmark and improves LongMemEval-S performance, while significantly reducing confabulation rates compared to prior systems. Ablation studies show that the generic lifecycle metadata drives correctness, whereas the behavioral ontology enhances calibration and reduces downstream hallucinations.

By Ansuman Mullick, Eray T\"uz\"un