arXiv:2606. 06564v1 Announce Type: cross Abstract: Residual connections are central to training deep Transformers, but standard PreNorm residual streams aggregate sublayer updates with fixed unit weights.
By Kehan Wang
LayerRoute is a parameter‑efficient technique that enables adaptive skipping of transformer layers in large language models. It adds a lightweight per‑layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters) to each of the 24 blocks in Qwen2.5‑0.5B‑Instruct, training them jointly with a gate‑regularized language‑modeling objective. Across ten independent runs, the method consistently identifies nine middle layers as skip‑eligible, achieves a verified wall‑clock speedup of 1.02x–1.06x, and improves perplexity by an average of 1.16 points, while the router’s decisions vary per input, confirming genuine adaptive behavior.
By Prateek Kumar Sikdar
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
By Tom\'as Figliolia, Beren Millidge
arXiv:2607. 09694v1 Announce Type: new Abstract: Attention Residuals replace the fixed residual sum with depthwise attention over previous sub-layer outputs in large language models (LLMs), but use each output as both a full-dimensional key and value.
By Jonathan Su
arXiv:2605. 20708v2 Announce Type: replace-cross Abstract: Diffusion Transformers (DiTs) have become a de facto backbone of modern visual generation, and nearly every major axis of their design -- tokenization, attention, conditioning, objectives, and latent autoencoders -- has been extensively revisited.
By Chao Xu, Maohua Li, Qirui Li, Yixuan Xu, Yanke Zhou, Yunhe Li, Cuifeng Shen, Hanlin Tang, Kan Liu, Tao Lan, Lin Qu, Shao-Qun Zhang
arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.
By Huzama Ahmad, Cao Viet Hai Nam, Se-Young Yun