← Back to all news
arXiv Machine Learning By Robert Jenkinson Alvarez

Prediction Is Not a Sufficient Memory State

Read the original on arXiv Machine Learning →

The paper shows that a memory state that reproduces every current prediction can still fail to learn from subsequent observations. It characterizes when a fixed‑dimensional continuous recurrent state exists for exact cumulative learning and attention, proving that for smooth convex residual losses the state exists only when the loss is polynomial. The work further quantifies the dimensionality required for various loss functions, demonstrates unbounded requirements for nonpolynomial losses, and analyzes specific cases such as ridge regression and kernel attention.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
6d ago

Exact Unlearning After Data Curation

The paper investigates what a service must retain to faithfully reproduce a training pipeline after data deletion requests, focusing on a rule where earlier similar records block later ones. It shows that a limit on future deletions precisely determines which records can still be needed for training, enabling an exact repair of training statistics after repeated deletions. The authors also construct finite‑precision features for ridge regression and demonstrate that the required private information can be minimized across a sequence of deletion requests, validating their approach on 1,001 AskUbuntu records.

By Priyank Jayraj, Devanshu sahoo, Uday Mittal
More like this →
arXiv Machine Learning
Aug 31

Fast Weight Attention for Continual Learning

Fast Weight Attention for Continual Learning introduces recurrent fast‑weight memories and selective state‑space models that compress expanding context into a fixed‑size recurrent state, enabling an online learning rule for state transitions. The paper derives normalized first‑order updates for squared‑error regression and negative inner‑product objectives, presenting several variants (Falcon‑1, Falcon‑2, Falcon‑3 and their inner‑product counterparts) with recurrent, masked‑parallel, and chunk‑parallel implementations. These methods demonstrate competitive performance in language modeling and improved length extrapolation on variable‑digit addition tasks.

By Yifan Zhang, Steve Ta, Jasper Zhang, Jichen Feng, Shuzhen Li, Yongxin Zhang, Yifeng Liu, Huizhuo Yuan, Mengdi Wang, Quanquan Gu, Andrew Chi-Chih Yao
safety
More like this →