Hugging Face Trending Papers October 5, 2026

The Optimization Landscape of Learning Compacted Context Models

Read the original on Hugging Face Trending Papers →

The paper investigates the challenges of optimizing compacted context models, particularly the KV cache, in continual learning scenarios. It identifies the optimization landscape as brittle and flat, and proposes a simplified Perceiver-based architecture that matches or surpasses full Perceiver transformers in continuous context compaction. Experiments on MCQ tasks in Finance, Legal, Gutenberg, and Code demonstrate the effectiveness of this approach.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv Machine Learning

Jun 9

Still: Amortized KV Cache Compaction in a Single Forward Pass

arXiv:2606. 07878v1 Announce Type: new Abstract: The KV cache is the memory bottleneck of long-horizon language model deployment.

By Charles O'Neill, Alex Sandomirsky, Harry Partridge, Mudith Jayasekara, Max Kirkby

llms nlp efficiency

arXiv Machine Learning

Jun 25

Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

arXiv:2606. 25342v1 Announce Type: new Abstract: Lifelong continual learning remains an obstacle on the path to human-like intelligence.

By Luke McDermott, Robert W. Heath jr., Rahul Parhi

llms agents efficiency

Hugging Face Trending Papers

Jun 8

End-to-End Context Compression at Scale

Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.

llms rag agents efficiency

Hugging Face Trending Papers

Jun 24

Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

Lifelong continual learning remains an obstacle on the path to human-like intelligence. Modern transformers show sparks of intelligence with in-context learning.

llms agents efficiency

arXiv AI

Jun 9

End-to-End Context Compression at Scale

arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.

By Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra, Zaiqian Chen, Artem Gazizov, Venkata Anoop Suhas Kumar Morisetty, Bhavya Kailkhura, Harshitha Menon, Zhuang Liu, Brian R. Bartoldson, Tom Goldstein, Sanae Lotfi, Micah Goldblum, Pavel Izmailov

llms rag agents efficiency

Hugging Face Trending Papers

Aug 20

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for fine-tuning models with sparse attention.

llms fine-tuning efficiency

One email a morning, machine-written

Related stories

Still: Amortized KV Cache Compaction in a Single Forward Pass

Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

End-to-End Context Compression at Scale

Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

End-to-End Context Compression at Scale

Learning how to Forget: Fine-tuning for Long-Context Sparse Attention