The paper revisits the interpretation of depth profiles in Diffusion Transformer backbones, showing that the commonly observed inverted‑U curve reflects the influence of the prediction target (e.g., ε, u, x0) rather than a simple shift from coarse to fine semantic granularity. It argues that the semantically meaningful signal is merely an intermediate by‑product of predicting the target, and that different training setups—particularly the choice of prediction target—predictably alter the curve’s shape. The authors demonstrate how this new understanding can guide the optimal placement of REPA modules and provide code for further exploration.
By Benyuan Meng, Longtao Huang, Yang Liu, Liang Lin
arXiv:2607. 13491v1 Announce Type: cross Abstract: Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters.
By Shuzhen Li, Yifan Zhang, Jiacheng Guo, Quanquan Gu, Mengdi Wang
arXiv:2605. 20708v2 Announce Type: replace-cross Abstract: Diffusion Transformers (DiTs) have become a de facto backbone of modern visual generation, and nearly every major axis of their design -- tokenization, attention, conditioning, objectives, and latent autoencoders -- has been extensively revisited.
By Chao Xu, Maohua Li, Qirui Li, Yixuan Xu, Yanke Zhou, Yunhe Li, Cuifeng Shen, Hanlin Tang, Kan Liu, Tao Lan, Lin Qu, Shao-Qun Zhang
The paper presents a theoretical framework for understanding generalization and scaling in Mixture-of-Experts (MoE) Transformers. It derives a sup‑norm covering‑number bound that separates active per‑input capacity from routing combinatorics, leading to a generalization bound under a manifold data model. Additionally, a constructive approximation theorem shows how error can be reduced by scaling active capacity or increasing the number of experts, from which neural scaling laws and compute‑optimal tradeoffs are derived.
By Mansour Zoubeirou a Mayaki
arXiv:2608. 02064v1 Announce Type: new Abstract: Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth.
By Timur Mudarisov, Mikhail Burtsev, Radu State
arXiv:2606. 18524v1 Announce Type: new Abstract: Looped (weight-tied) Transformers apply a shared residual block $N$ times ($h \leftarrow h + \varepsilon\,f(h)$, same $f$ at each step), increasing effective depth without adding parameters.
By Shaowen Wang, Bingrui Li, Ge Zhang, Wenhao Huang, Shen Yan, Jian Li