arXiv:2608. 04448v1 Announce Type: cross Abstract: Diffusion transformers deliver strong image generation, but their training cost grows superlinearly with resolution.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
The paper introduces the Drift Contract, a spectral update geometry for local learning that improves depth robustness and hyperparameter stability. By applying momentum orthogonalization with spectral step scaling to per‑layer updates, the authors achieve consistent performance across a wide range of widths and depths on CIFAR‑10 MLPs, outperforming local Adam and providing a per‑layer, input‑conditioned drift bound. The study also shows that the spectral geometry itself, rather than step‑size rules, drives the observed depth robustness, while a negative result indicates that the stability benefit is limited to non‑normalized layers.
arXiv:2607. 23711v1 Announce Type: new Abstract: LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix $W+BA$ that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting.
arXiv:2608. 04084v1 Announce Type: new Abstract: Mixture-of-experts (MoE) networks pursue specialization through learned routers, gates, and load-balancing losses, yet at matched total-parameter budgets learned routers can underperform equal-weight No-Routing baselines.
The paper investigates how the usefulness of training samples, as determined by coreset selection, depends on the learner rather than just the data. Experiments on ImageNet-100 and ImageNet-1k show that changing model width, input grid, stride, and architecture (e.g., ResNet vs. ViT) shifts the crossover point where different selection criteria (easy-first vs. geometric coverage) become optimal. These findings demonstrate that the relative value of a fixed subset of samples varies with the target learner’s capacity and structure, and that selection strategies must be tuned to the specific model they will train.
The paper revisits the interpretation of depth profiles in Diffusion Transformer backbones, showing that the commonly observed inverted‑U curve reflects the influence of the prediction target (e.g., ε, u, x0) rather than a simple shift from coarse to fine semantic granularity. It argues that the semantically meaningful signal is merely an intermediate by‑product of predicting the target, and that different training setups—particularly the choice of prediction target—predictably alter the curve’s shape. The authors demonstrate how this new understanding can guide the optimal placement of REPA modules and provide code for further exploration.
arXiv:2605. 17109v3 Announce Type: replace-cross Abstract: In recent years, Muon has emerged as the dominant method for training large language models, and transformers more broadly.