arXiv Machine Learning
Aug 11
FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients
arXiv:2608. 09250v1 Announce Type: new Abstract: Federated learning (FL) must serve devices with varying computational capabilities.
ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer
Related stories
arXiv Machine Learning
Oct 19
HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models
arXiv:2610.23455v1 Announce Type: cross Abstract: LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fix...
arXiv Machine Learning
Jun 29
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.
arXiv AI
Aug 21
Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts
arXiv:2608. 20061v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.
arXiv Machine Learning
Sep 10
Dense Structural Compression of Transformers via Gauge-Correct Channel Removal
The paper introduces GaugeLasso, a method that applies symmetric group‑lasso penalties to transformer channels during training, enabling entire tensor slices to be zeroed out while maintaining dense tensors for GPU efficiency. By calibrating channel penalties based on inference utility per compute, the network self‑organizes into depth‑dependent structural profiles that can be dramatically smaller than the original architecture, achieving up to 255‑fold compression on a polynomial division task and outperforming hand‑designed baselines on language modeling and autoencoding benchmarks. The approach also accelerates training and reveals over‑provisioned axes that guide subsequent design iterations.
arXiv Statistics ML
Oct 2
Transferable Graph Metanetworks
Transferable Graph Metanetworks extend the graph metanetwork paradigm to predict properties of neural networks of varying widths. By enforcing invariance to width-dependent representations and continuity across similar functions, the proposed modifications enable robust size generalization, especially for networks trained under the maximal‑update parameterization (μP) up to 42× the training width. The authors provide empirical evidence across multiple tasks and theoretical guarantees in the infinite‑width limit, explaining both successes and limitations under different parameterizations.
arXiv AI
Jun 16
Scaling Adaptive Depth with Norm-Agnostic Residual Networks
arXiv:2606. 16112v1 Announce Type: cross Abstract: Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth.
arXiv Machine Learning
Jun 25
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.
arXiv Machine Learning
Sep 25
Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling
Spectral-Guided Diffusion introduces a method to accelerate diffusion inference by identifying and reusing residual branches that need not be recomputed during the trajectory. The approach uses a Spectral Concentration Ratio (SCR) combined with Frobenius magnitude to create an offline sensitivity proxy and deterministic lifetime for each scheduled unit, eliminating the need for routers or input-dependent searches. Experiments on models such as LLaDA-8B, DiT-XL/2, U-ViT-L, and SDXL show that this scheduling preserves quality better than several baselines and achieves up to a 3.0× wall‑clock speedup over eager inference.
arXiv AI
Oct 27
Fantastic Pretraining Optimizers and Where to Find Them
The paper evaluates ten deep learning optimizers across models ranging from 0.1B to 1.2B parameters and varying data-to-model ratios, highlighting that fair comparisons require rigorous hyperparameter tuning and end-of-training evaluations. It finds that claimed speedups over AdamW are lower than advertised, dropping to only 1.1× for the largest models, and that intermediate checkpoint comparisons can mislead due to learning‑rate decay. The study shows that matrix‑based optimizers like Muon and Soap achieve the highest speedups, but these benefits diminish as model size increases.
arXiv AI
Jul 3
$\mu$pscaling small models: Principled warm starts and hyperparameter transfer
arXiv:2602. 10545v2 Announce Type: replace-cross Abstract: Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets.
arXiv Machine Learning
1d ago
Module-Aware Optimization for Graph Neural Networks
The paper introduces GraphMuon, a graph-aware optimizer that extends the matrix-aware Muon optimizer by incorporating the covariance structure of graph‑propagated layer inputs. GraphMuon performs orthogonalization in a metric defined by these propagated representations and normalizes step sizes based on the update’s effect on the layer output, thereby mitigating graph‑induced update amplification. Empirical results show that GraphMuon achieves 1.32×–2.42× faster validation‑loss time‑to‑target compared to AdamW across 15 dataset‑architecture configurations on large‑scale OGB benchmarks, while maintaining competitive predictive performance.