arXiv Computer Vision
Compressing History into Memory: Distilling Transformers into Recurrent Transformers
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv Machine Learning
Aug 19
Dynamic Compression in Recurrent Networks
Dynamic Compression in Recurrent Networks proposes a method that lets recurrent models revisit and revise their fixed-size state through additional updates, rather than compressing all information in a single causal pass. This approach allows the model to retain lower-fidelity history and refine only the relevant parts when needed, reducing the required state size for accurate task reuse. Experiments show that dynamic compression lowers the recurrent state needed and scales better as the number of stored functions increases.
arXiv Machine Learning
Jun 16
Next-Latent Prediction Transformers Learn Compact World Models
arXiv:2511. 05963v4 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens.
arXiv Computer Vision
2d ago
FlashBack: Knowing When to Remember in Streaming Vision-Language Models
arXiv:2610.01192v1 Announce Type: new Abstract: Streaming vision-language models must process continuously growing video streams under a bounded compute budget, creating a persistent tension between...
arXiv Computation and Language
Sep 16
Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization
The paper proposes a lightweight recurrent memory module inserted between the lower and upper halves of a 6‑layer decoder‑only transformer. This module, which uses cross‑attention to observe hidden states, a GRU to update a persistent state, and gated addition to modulate subsequent layers, adds only 3.7% more parameters. It reduces evaluation loss by 28.5% and narrows the generalization gap, with ablations showing the benefit comes solely from the memory topology rather than auxiliary losses.
Hugging Face Trending Papers
Aug 18
Dynamic Compression in Recurrent Networks
Dynamic Compression in Recurrent Networks proposes a method for recurrent models to selectively revisit and update past tokens, rather than compressing all history in a single causal pass. By allowing the model to refine its fixed-size state only when needed, it can maintain lower-fidelity information in the raw sequence and revisit it later. Experiments show that this selective re-scanning reduces the recurrent state needed for accurate task reuse and scales better as the number of stored functions increases.
arXiv AI
4d ago
Compress to Remember: Learning Compact Memory via On-Policy Distillation for Long Video Generation
The paper introduces Prediction‑Aligned Context Compaction (PACC), a method that learns a compact memory representation for long‑video generation by distilling a frozen video generator. PACC trains a compressor to aggregate past frames into memory tokens, using the generator as both teacher and student during on‑policy distillation. Experiments on MBench and VBench‑Long show that PACC improves memory‑event coverage and consistency, achieving better scores than strong baselines and producing competitive minute‑long videos.