arXiv Machine Learning
Memory by Design: Bayesian Layers for Sequence Modeling
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 11
Memory by Design: Probabilistic Sequence Layers
arXiv:2605. 31163v2 Announce Type: replace-cross Abstract: We introduce the design-model framework: a way to derive efficient recurrent sequence maps from explicit assumptions about memory.
arXiv AI
Oct 15
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
arXiv AI
Sep 10
Kalman Delta Networks: Uncertainty-aware Associative Memory
Kalman Delta Networks (KDNs) extend linear attention models by treating associative memory as a linear–Gaussian state‑space system, enabling the Kalman filter to optimally estimate both memory state and its uncertainty. Two GPU‑friendly approximations—Diagonal KDN and Isotropic KDN—use mean‑field variational inference or a single scalar uncertainty per head, respectively, to maintain tractable uncertainty recurrences during linear‑attention scans. Experiments on 750 M and 1.3 B‑parameter models show that KDN variants consistently lower perplexity and raise downstream accuracy compared to existing linear‑attention baselines.
arXiv Machine Learning
Sep 16
Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks It
arXiv:2609.16183v1 Announce Type: new Abstract: Fixed-state recurrences--linear attention and state-space models--are reported to lag behind attention on associative recall, but whole-architecture co...
arXiv Machine Learning
Aug 31
Attention as Conditioning: What Classical Learning Theory Predicts About Linear Transformers
arXiv:2508. 08289v3 Announce Type: replace Abstract: Attention is widely understood as an associative memory, but that description alone does not predict how the memory will behave.
arXiv Machine Learning
Aug 31
DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging
The paper introduces DARTS, a method for tuning decoder representations during model merging. It addresses representation bias in autoregressive decoders by using an entropy‑weighted L1 loss and a per‑position additive bias to correct errors that accumulate across token positions. Experiments on code generation, mathematical reasoning, and instruction following with Llama‑2‑7B show that DARTS improves performance over standard surgery while adding only 0.1% extra parameters.