arXiv Machine Learning June 30, 2026 By Samat Zharassov

SamatNext v0.2-B: An Exploratory Study of RMS-Normalized Hybrid Decoders for Curriculum Retention in Small Code Models

Read the original on arXiv Machine Learning →

arXiv:2606. 22248v2 Announce Type: replace Abstract: Standard autoregressive Transformer decoders can often exhibit substantial forgetting under sequential fine-tuning on shifting curriculum distributions.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 18

Towards Understanding What State Space Models Learn About Code

arXiv:2602. 06774v2 Announce Type: replace Abstract: State Space Models (SSMs) have emerged as an efficient alternative to the Transformer architecture.

By Jiali Wu, Abhinav Anand, Shweta Verma, Mira Mezini

llms fine-tuning

arXiv Machine Learning

Jul 14

Controllably Efficient Language Models

arXiv:2511. 05313v2 Announce Type: replace Abstract: The substantial inference costs of attention in transformers motivated the development of efficient sequence mixers: namely sparse and sliding window attention, convolutions and linear attention.

By Jatin Prakash, Aahlad Puli, Rajesh Ranganath

llms efficiency benchmarks

arXiv Machine Learning

Jul 30

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

arXiv:2505. 15548v2 Announce Type: replace Abstract: Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic.

By Suvadeep Hajra

llms

arXiv AI

Aug 11

Full-bandwidth transformer

arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.

By Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford

llms rag efficiency

arXiv Machine Learning

Jun 29

Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing

arXiv:2606. 27449v1 Announce Type: new Abstract: Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension (dh = dmodel/h) throughout the models depth.

By Shubham Aggarwal

llms benchmarks

arXiv Machine Learning

Jun 11

On Subquadratic Architectures: From Applications to Principles

arXiv:2606. 12364v1 Announce Type: new Abstract: Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost.

By Anamaria-Roberta Hartl, Levente Z\'olyomi, David Stap, Pieter-Jan Hoedt, Niklas Schmidinger, Lukas Hauzenberger, Sebastian B\"ock, G\"unter Klambauer, Sepp Hochreiter

llms efficiency

Related stories

Towards Understanding What State Space Models Learn About Code

Controllably Efficient Language Models

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

Full-bandwidth transformer

Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing

On Subquadratic Architectures: From Applications to Principles