arXiv AI
Jun 24
Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs
arXiv:2511. 20892v4 Announce Type: replace Abstract: Large language models (LLMs) often produce incorrect or outdated content after being employed.
Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory
arXiv:2303. 01421v2 Announce Type: replace-cross Abstract: Semiparametric language models (LMs) have shown promise in various Natural Language Processing (NLP) tasks.
Related stories
arXiv AI
Aug 11
Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages
arXiv:2603. 12658v2 Announce Type: replace-cross Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs.
arXiv Machine Learning
5d ago
LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining
arXiv:2608. 12419v1 Announce Type: new Abstract: Large language models (LLMs) have achieved remarkable breakthroughs across various applications.
arXiv AI
Jun 16
Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift
arXiv:2606. 15734v1 Announce Type: cross Abstract: Continual post-training enables models to absorb emerging knowledge after deployment, but repeatedly updating shared parameters can accumulate weight drift, potentially causing catastrophic forgetting and degrading general capabilities.
arXiv Machine Learning
Jun 5
Pretraining Recurrent Networks without Recurrence
arXiv:2606. 06479v1 Announce Type: new Abstract: Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations.
arXiv AI
Jun 4
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.
arXiv AI
Jul 24
Scaling Interpretable Transformers with Parity Bottleneck Layers
arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.
arXiv AI
23h ago
Cross-Model Memory Transfer via Target-Side Reader Adaptation
arXiv:2608. 17050v1 Announce Type: cross Abstract: Methods for improving knowledge use in large language models typically fall into two regimes.
arXiv Machine Learning
Jun 5
A Survey on Diffusion Language Models
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
arXiv Machine Learning
Jun 3
Dynamic Short Convolutions Improve Transformers
arXiv:2606. 03825v1 Announce Type: new Abstract: Transformers have become the dominant architecture for large language models, largely due to the scalability and flexibility of attention, feed-forward layers, residual connections, and normalization.
arXiv AI
Jun 2
Learning to Remember, Learn, and Forget in Attention-Based Models
arXiv:2602. 09075v3 Announce Type: replace-cross Abstract: In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks.
arXiv Machine Learning
Jun 25
Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
arXiv:2606. 25342v1 Announce Type: new Abstract: Lifelong continual learning remains an obstacle on the path to human-like intelligence.