arXiv:2511. 04981v2 Announce Type: replace Abstract: Model depth is a double-edged sword in deep learning: deeper models achieve higher accuracy but require higher computational cost.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
arXiv:2602. 07494v2 Announce Type: replace Abstract: Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning.
arXiv:2602. 10545v2 Announce Type: replace-cross Abstract: Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets.
The paper demonstrates that layer dropout, also known as stochastic depth, can be effectively used in state‑of‑the‑art large language model (LLM) training. By optimizing the layer distribution, schedule, and optimizer settings, the authors show that layer dropout can reduce training loss while saving up to 25 % of training FLOPs. Additionally, layer dropout enables post‑training optimizations such as early exit and self‑speculative decoding, achieving up to 1.5× inference speedup with negligible accuracy loss across models ranging from 271 M to 8.2 B parameters and datasets up to 160 B tokens.
The paper proposes a new strategy for merging layers in deep neural networks, enabling depth compression without requiring an analytical solution for convolutions with padding and without increasing kernel size. This approach addresses limitations of previous methods that struggled with padded convolutions and larger kernels, and it is validated across various architectures and datasets with measured inference speed-ups on embedded platforms.
arXiv:2402.11215v4 Announce Type: replace Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-sc...