arXiv:2606. 04048v1 Announce Type: cross Abstract: Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods.
arXiv:2604.19021v3 Announce Type: replace Abstract: Linear attention mechanisms have emerged as promising alternatives to softmax attention, offering linear-time complexity during inference. Recent a...
arXiv:2607. 05017v1 Announce Type: cross Abstract: The performance of deep learning models crucially depends on the settings of hyperparameters like learning rate, initialization scale, and weight decay.
arXiv:2607. 07706v1 Announce Type: new Abstract: The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference.
arXiv:2601. 00417v4 Announce Type: replace-cross Abstract: Transformer residual streams evolve through additive updates.
arXiv:2602. 10545v2 Announce Type: replace-cross Abstract: Modern large-scale neural networks are often trained and released in multiple sizes to accommodate diverse inference budgets.
arXiv:2602. 07494v2 Announce Type: replace Abstract: Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning.
arXiv:2603. 07523v3 Announce Type: replace Abstract: Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales.
The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality.
arXiv:2606. 31717v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks.
arXiv:2606. 00888v1 Announce Type: cross Abstract: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.
arXiv:2610. 01172v1 Announce Type: new Abstract: We study learning rate (LR) scaling for hybrid architectures combining Transformer and State-Space Model (SSM) blocks, a class adopted by several recent production language models.
Neural Cellular Automata (NCAs) are shown to learn general, scale‑invariant topological primitives in their hidden channels, which can be transferred from a teacher to a student model for few‑shot learning. The study introduces a transfer‑learning mechanism that injects pretrained hidden states into a student, improving early optimization and outperforming recurrent and feed‑forward baselines on MNIST benchmarks with only ~9,800 parameters. Mechanistic analysis reveals that hidden channels decouple feature extraction from classification, converging to mutually orthogonal states that absorb morphological complexity.