arXiv Machine Learning June 16, 2026 By Zhongzhu Zhou, Qingyang Wu, Junxiong Wang, Mayank Mishra, Shuaiwen Leon Song, Ben Athiwaratkun, Chenfeng Xu

Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

Read the original on arXiv Machine Learning →

arXiv:2606. 16429v1 Announce Type: new Abstract: Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jul 7

Effective Distillation to Hybrid xLSTM Architectures

arXiv:2603. 15590v2 Announce Type: replace Abstract: There have been numerous attempts to distill quadratic attention-based large language models (LLMs) into sub-quadratic linearized architectures.

By Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied, Anamaria-Roberta Hartl, David Stap, Pieter-Jan Hoedt, Maximilian Beck, Sebastian B\"ock, G\"unter Klambauer, Sepp Hochreiter

llms efficiency

arXiv Machine Learning

Sep 10

Conditioned Initialization for Attention

arXiv:2609.07086v1 Announce Type: new Abstract: Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their su...

By Hemanth Saratchandran, Simon Lucey

llms safety

arXiv AI

Jul 20

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

arXiv:2603. 26556v2 Announce Type: replace-cross Abstract: Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs.

By Juan Gabriel Kostelec, Qinghai Guo

llms efficiency benchmarks

arXiv AI

Jun 3

Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction

arXiv:2601. 11667v2 Announce Type: replace-cross Abstract: Transformer architectures deliver state-of-the-art accuracy via dense full-attention, but their quadratic time and memory complexity with respect to sequence length limits practical deployment.

By Xiaojie Xia, Huigang Zhang, Chaoliang Zhong, Jun Sun, Yusuke Oishi

llms efficiency benchmarks

arXiv Machine Learning

2d ago

Activation-Conditioned Self-Distillation

arXiv:2609.38342v1 Announce Type: new Abstract: On-policy self-distillation uses a model as its own teacher to provide dense supervision for reasoning, often through reference-solution conditioning....

By Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu

efficiency benchmarks

arXiv AI

Jul 23

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.

By Yu Zhao, Zekun Zhang, Fan Jiang, Bo Zeng, Linlong Xu, Shimin Shan, Yu Liu, Longyue Wang, Weihua Luo

efficiency benchmarks

One email a morning, machine-written

Related stories

Effective Distillation to Hybrid xLSTM Architectures

Conditioned Initialization for Attention

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

Distill-then-Replace: Efficient Task-Specific Hybrid Attention Model Construction

Activation-Conditioned Self-Distillation

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning