arXiv Machine Learning July 24, 2026 By Yidu Wu, Xiang Wang, Kejie Zhao, Zhangchi Wang, Qinghai Guo, Xiaoying Tang

Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

Read the original on arXiv Machine Learning →

arXiv:2607. 21291v1 Announce Type: cross Abstract: Large language models (LLMs) achieve strong generation and reasoning performance, but the Transformer architecture incurs high inference cost.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Aug 27

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

The paper introduces Gated Recurrent Transformers, a depth‑sharing architecture that brackets a single shared core with fixed prelude and coda blocks and uses a lightweight projection and element‑wise update gate to modulate recurrent updates. This design allows functional specialization across recurrences while reducing memory footprint. Experiments show that, under equal FLOPs or parameter budgets, the recurrent model matches or surpasses deeper GPT‑2 Small baselines, achieving similar or better accuracy with fewer parameters and lower peak decoding memory.

By Amr Hegazy, Amr Alanwar, Mostafa Elhoushi

llms efficiency

arXiv Machine Learning

Aug 18

RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers

arXiv:2608. 15062v1 Announce Type: cross Abstract: Scaling transformer language models creates an inherent tension between expressivity and memory efficiency.

By Amr Hegazy, Amr Alanwar, Mostafa Elhoushi

llms efficiency

arXiv AI

Aug 14

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

arXiv:2608. 13426v1 Announce Type: cross Abstract: Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications.

By Zixuan Lan, Yanhong Li, Jiawei Zhou

llms multimodal benchmarks

arXiv Machine Learning

Aug 26

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation in Transformers

arXiv:2608.15062v3 Announce Type: replace-cross Abstract: Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layer...

By Amr Hegazy, Amr Alanwar, Mostafa Elhoushi

llms efficiency

arXiv AI

Jun 6

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

arXiv:2606. 05173v1 Announce Type: cross Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure.

By Aimen Boukhari

rag diffusion benchmarks

arXiv Machine Learning

Sep 1

You Do Not Fully Utilize Transformer's Representation Capacity

arXiv:2502.09245v3 Announce Type: replace Abstract: In contrast to RNNs, which compress their history into a single hidden state, Transformers can attend to all past tokens directly. However, standar...

By Gleb Gerasimov, Yaroslav Aksenov, Nikita Balagansky, Viacheslav Sinii, Daniil Gavrilov

llms

One email a morning, machine-written

Related stories

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation in Transformers

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

You Do Not Fully Utilize Transformer's Representation Capacity