arXiv AI September 24, 2026 By Zhiheng Hu, Yixun Wei, Jian Zhou, Yizhuang Zhou, Ji Li, Xing Chen, Yang Li, Bojun Wang, Yibo Zhu, Xiangyu Zhang, Daxin Jiang

KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling

Read the original on arXiv AI →

KITE (KV-Invariant Transformer Expansion) is a scaling paradigm that trains a language model from a smaller size to a larger one, saving training costs by upcycling. It places new parameters in regions that do not affect attention KV, so inference only requires prefilling KV from the smaller part, reducing inference costs. The Step Scale Transformer (SST), a two-tower decoder, demonstrates this by achieving lower training loss than comparable MoE Transformers while cutting estimated inference cost by 6.7% and 31.6%.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 28

Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory

arXiv:2605. 09877v4 Announce Type: replace-cross Abstract: Recall presents a difficult choice: transformers have a linearly growing memory that slows each successive token, while linear RNNs typically have fixed costs but limited recall.

By Daniel Goldstein, Navneel Singhal, Eugene Cheah

llms

arXiv AI

Aug 18

Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation

arXiv:2608. 12385v2 Announce Type: replace Abstract: As large language models serve ever more requests, cumulative inference cost is growing relative to the one-time cost of training.

By Liming Liu, Mingze Wang, Tuo Zhao

llms rag efficiency

arXiv Computation and Language

Sep 10

Revisiting the Shape Convention of Transformer Language Models

arXiv:2602.06471v2 Announce Type: replace Abstract: The architectural shape of dense Transformers has remained remarkably stable: narrow-wide-narrow feed-forward networks (FFNs) consume most non-embe...

By Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin, Meng-Hsi Chen, Da-shan Shiu

llms rag

Hugging Face Trending Papers

Jul 9

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.

llms rag

arXiv AI

Sep 1

A Universal Context-Reuse Layer for Cross-Model KV Sharing

arXiv:2608.30963v1 Announce Type: cross Abstract: Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own pre...

By Yi Li, Dongming Jiang, Yi Zhao, Bingzhe Li

llms agents nlp

arXiv AI

Jun 16

PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression

arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.

By Chao Fei, Panos Kalnis

llms efficiency

One email a morning, machine-written

Related stories

Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory

Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation

Revisiting the Shape Convention of Transformer Language Models

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

A Universal Context-Reuse Layer for Cross-Model KV Sharing

PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression