arXiv AI July 1, 2026 By Duc Anh Nguyen, Tien Ngoc Luu, Tung Pham, Toan Tran

Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models

arXiv:2606. 31397v1 Announce Type: cross Abstract: State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Aug 11

Full-bandwidth transformer

arXiv:2608. 08888v1 Announce Type: new Abstract: Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth.

By Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford

llms rag efficiency

arXiv AI

Aug 11

MixFormer: Linear Transformer with Mixture of Memory Experts

arXiv:2608. 09468v1 Announce Type: cross Abstract: State Space Models (SSMs), as a mainstream research direction of linear Transformers, aim to achieve higher efficiency than standard Transformers in long-context modeling.

By Yu Guo, Lei Duan

llms diffusion

arXiv Machine Learning

Jun 5

SpanNorm: Reconciling Training Stability and Performance in Deep Transformers

arXiv:2601. 22580v2 Announce Type: replace-cross Abstract: The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures.

By Chao Wang, Bei Li, Jiaqi Zhang, Xinyu Liu, Yuchun Fan, Linkun Lyu, Xin Chen, Jingang Wang, Tong Xiao, Peng Pei, Xunliang Cai

llms

arXiv Machine Learning

Jul 1

Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

arXiv:2606. 31717v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks.

By Shuai Yuan, Sudong Cai, Bingzhi Chen, Shuyuan Zheng, Chuan Xiao, Makoto Onizuka, Rui Mao

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition

arXiv:2411. 09816v5 Announce Type: replace Abstract: Large neural networks achieve state-of-the-art performance on many tasks, yet their sheer size hinders deployment on resource-constrained devices.

By Cem \"Uy\"uk, Mike Lasby, Mohamed Yassin, Utku Evci, Yani Ioannou

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jul 7

Equivalence of Context and Parameter Updates in Modern Transformer Blocks

arXiv:2511. 17864v3 Announce Type: replace Abstract: Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its MLP weights.

By Adrian Goldwaser, Michael Munn, Javier Gonzalvo, Benoit Dherin

llms

Related stories

Full-bandwidth transformer

MixFormer: Linear Transformer with Mixture of Memory Experts

SpanNorm: Reconciling Training Stability and Performance in Deep Transformers

Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition

Equivalence of Context and Parameter Updates in Modern Transformer Blocks