arXiv AI
Aug 11
Full-bandwidth transformer
arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
WhiteMatter: All-to-All Cross-Layer Connections via KV Mixing
WhiteMatter introduces a novel architecture for Transformers that connects every attention layer to representations from all layers of each past token, allowing connection weights to vary across consumer layers and adapt to the source token. The design uses a router to mix the $L$ layer states of each token into $k$ KV channels, which are cached for subsequent tokens; each consumer layer attends to one channel. Experiments show that WhiteMatter outperforms a vanilla Transformer with 50% more layers and maintains most of this advantage even when the KV-cache is compressed by 50%.
Related stories
arXiv Machine Learning
Sep 24
Shared Global KV with Layer-Specific Local History
The paper investigates how to combine shared global key‑value (KV) caches with layer‑specific local history in decoder‑only Transformer language models. By separating historical content from the input source used to form it, the authors show that adding local history can reduce held‑out test perplexity by about 1.4% compared to a current‑token local branch, while also demonstrating benefits in capacity, entry‑count, and training‑compute controls. Experiments on a 126M‑parameter model with 2K context reveal that local history remains valuable even when adjacent layers share local inputs, and that a sufficient suffix schedule can reduce upper‑layer construction work without losing cache completeness.
arXiv AI
Aug 11
RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation
arXiv:2608. 08684v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging.
arXiv Machine Learning
Jun 5
You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
arXiv:2606. 06467v1 Announce Type: cross Abstract: Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought.
arXiv Machine Learning
Sep 1
You Do Not Fully Utilize Transformer's Representation Capacity
arXiv:2502.09245v3 Announce Type: replace Abstract: In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standar...
Hugging Face Trending Papers
Sep 23
Shared Global KV with Layer-Specific Local History
Decoder-only Transformer language models cache keys and values (KV) to reuse past computation during generation. Sharing KV across layers saves storage but reduces the diversity of representations ava...
arXiv AI
5d ago
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
arXiv:2609.32259v2 Announce Type: replace Abstract: Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires ea...
arXiv AI
Jun 16
PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression
arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.
arXiv AI
Jul 28
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
arXiv:2605. 09877v4 Announce Type: replace-cross Abstract: Recall presents a difficult choice: transformers have a linearly growing memory that slows each successive token, while linear RNNs typically have fixed costs but limited recall.
arXiv AI
Jul 8
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
arXiv:2607. 06523v1 Announce Type: new Abstract: Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation.
arXiv AI
1d ago
MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference
arXiv:2609.34077v2 Announce Type: replace-cross Abstract: Mixture-of-experts (MoE) language models often exceed the memory of a single GPU. Expert offloading keeps most experts in host memory and loa...
arXiv Machine Learning
Jun 9
Latent Cache Flow: Model-to-Model Communication Without Text
arXiv:2605. 22863v2 Announce Type: replace Abstract: LLM agents today communicate via text, which incurs considerable latency and information loss due to the need to autoregressively decode the sharer model's state and encode at the receiver model.