arXiv AI
Aug 11
Full-bandwidth transformer
arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.
WhiteMatter: All-to-All Cross-Layer Connections via KV Mixing
WhiteMatter introduces a novel architecture for Transformers that connects every attention layer to representations from all layers of each past token, allowing connection weights to vary across consumer layers and adapt to the source token. The design uses a router to mix the $L$ layer states of each token into $k$ KV channels, which are cached for subsequent tokens; each consumer layer attends to one channel. Experiments show that WhiteMatter outperforms a vanilla Transformer with 50% more layers and maintains most of this advantage even when the KV-cache is compressed by 50%.
Related stories
arXiv AI
Aug 11
RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation
arXiv:2608. 08684v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging.
arXiv Machine Learning
Jun 5
You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
arXiv:2606. 06467v1 Announce Type: cross Abstract: Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought.
arXiv AI
Jun 16
PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression
arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.
arXiv AI
Jul 28
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
arXiv:2605. 09877v4 Announce Type: replace-cross Abstract: Recall presents a difficult choice: transformers have a linearly growing memory that slows each successive token, while linear RNNs typically have fixed costs but limited recall.
arXiv AI
Jul 8
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
arXiv:2607. 06523v1 Announce Type: new Abstract: Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation.
arXiv Machine Learning
Jun 9
Latent Cache Flow: Model-to-Model Communication Without Text
arXiv:2605. 22863v2 Announce Type: replace Abstract: LLM agents today communicate via text, which incurs considerable latency and information loss due to the need to autoregressively decode the sharer model's state and encode at the receiver model.
arXiv AI
4d ago
Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation
arXiv:2608. 12385v2 Announce Type: replace Abstract: As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training.
arXiv Machine Learning
Jul 2
When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaboration
arXiv:2604. 13349v2 Announce Type: replace Abstract: Communication in Large Language Model (LLM)-based multi-agent systems is moving beyond discrete tokens to preserve richer context.
arXiv Machine Learning
Jul 31
Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.
arXiv AI
Jun 26
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.
arXiv Machine Learning
Jun 9
BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference
arXiv:2606. 09514v1 Announce Type: new Abstract: Large language models (LLMs) incur high inference cost due to their depth and parameter scale.