The paper investigates why Diffusion Transformers (DiT) produce weaker features for vision tasks compared to their strong generative performance. Through controlled training, the authors demonstrate that incorporating long skip connections (LSCs) significantly improves feature quality. They explain that LSCs allow diffusion noise to bypass the main backbone, preventing noise pollution of image signals and aligning the architecture more closely with the diffusion objective.
arXiv:2605. 20708v2 Announce Type: replace-cross Abstract: Diffusion Transformers (DiTs) have become a de facto backbone of modern visual generation, and nearly every major axis of their design -- tokenization, attention, conditioning, objectives, and latent autoencoders -- has been extensively revisited.
arXiv:2610.07002v1 Announce Type: new Abstract: Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides fea...
The paper introduces SVDtrunc, a two‑step block‑level compression method for Diffusion Transformers (DiTs) that allocates ranks across blocks, applies truncated SVD to the least important ones, and then fine‑tunes all blocks with modular knowledge distillation and a rectified‑flow objective. Experiments on FLUX.dev show that SVDtrunc achieves near‑full performance at 68% of the original parameters and remains competitive even at 57%, outperforming all competing approaches on GenEval, HPSv2, and DPG benchmarks. The method also works well without fine‑tuning, complementing step distillation and offering a practical path to efficient large‑scale generative models.
The paper investigates why Diffusion Transformers perform well when predicting clean data but poorly when predicting noise or velocity. It introduces the concept of *residual-stream burden*, explaining that noisy targets force the residual stream to carry noise-dependent variation through the network, limiting representation flexibility. Experiments show that spectral concentration of clean targets reduces this burden, and the authors propose Spatially Indexed Hyper-Connections (SiHC) to expand residual bandwidth, achieving state‑of‑the‑art FID on ImageNet.
arXiv:2609.37080v1 Announce Type: new Abstract: Latent Diffusion Models (LDMs) typically adopt a two-stage pipeline: an auto-encoder (AE) is first pre-trained to define a latent space, then a diffusi...
arXiv:2606. 15796v1 Announce Type: cross Abstract: Mechanistic interpretability seeks to explain neural network behavior by decomposing model computations into interpretable features and circuits.
The paper revisits the interpretation of depth profiles in Diffusion Transformer backbones, showing that the commonly observed inverted‑U curve reflects the influence of the prediction target (e.g., ε, u, x0) rather than a simple shift from coarse to fine semantic granularity. It argues that the semantically meaningful signal is merely an intermediate by‑product of predicting the target, and that different training setups—particularly the choice of prediction target—predictably alter the curve’s shape. The authors demonstrate how this new understanding can guide the optimal placement of REPA modules and provide code for further exploration.
arXiv:2605. 13974v2 Announce Type: replace-cross Abstract: Diffusion Transformers (DiTs) and related flow-based architectures are now among the strongest text-to-image generators, yet the internal mechanisms through which prompts shape image semantics remain poorly understood.
The paper investigates how diffusion transformers (DiTs) denoise images by framing them as aggregator–predictor models that use context aggregation and patch prediction. It shows that for Gaussian Markov random field priors, a global Bayes denoiser can be approximated with a limited context, and that patchwise linear denoisers recover empirical local denoisers when aggregation is informative. Experiments confirm that DiTs trained on real images exhibit noise‑dependent denoising contexts and that replacing DiT MLPs with shared‑memory patch predictors yields competitive performance, while cross‑dataset component transplantation supports the proposed functional roles of attention and MLPs.
Recent advances in Diffusion Transformers (DiTs) have enabled remarkable progress in visual synthesis, benefiting from their superior scalability. To facilitate DiTs' capability of capturing meaningful internal representations, recent works such as REPA incorporate external pretrained encoders for representation alignment.
The paper investigates how diffusion transformer acceleration methods such as REPA and dispersive regularization affect the AdaLN‑Zero residual gate α, which controls conditioned updates in the residual stream. It finds that these methods reorganize α, making accelerated models more sensitive to its perturbations and aligning gate geometry with semantic class relationships. Introducing a normalized participation ratio (nPR) to measure gate channel organization, the authors show that broader gate participation correlates with higher generation quality and propose a teacher‑free radial objective that encourages this broader participation, improving FID on ImageNet.
Latent diffusion models achieve strong generative performance by operating in a compressed latent space produced by a variational autoencoder (VAE). However, it remains unclear whether all latent channels contribute equally to the diffusion process, or whether significant redundancy exists.