arXiv Machine Learning

p-Spin Glass Network Efficient Single-Batch Continual Learning

arXiv:2608. 14774v1 Announce Type: new Abstract: Modern sequence models heavily rely on massive memory footprints and large-batch stochastic optimization, barriers that restrict sample efficiency and continual learning.

arXiv AI

Jun 2

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

arXiv:2606. 00888v1 Announce Type: cross Abstract: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.

By Qiao Xiao, Boqian Wu, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Elena Mocanu, Mykola Pechenizkiy, Decebal Constantin Mocanu, Torsten Hoefler

llms efficiency

arXiv Machine Learning

Jul 21

Chameleon: A Multiplier-Free Temporal Convolutional Network Accelerator for End-to-End Few-Shot and Continual Learning from Sequential Data

arXiv:2505. 24852v3 Announce Type: replace-cross Abstract: On-device learning at the edge enables low-latency, private personalization with improved long-term robustness and reduced maintenance costs.

By Douwe den Blanken, Charlotte Frenkel

benchmarks

arXiv AI

Jun 9

Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications

arXiv:2605. 11855v2 Announce Type: replace-cross Abstract: Sequence learning is dominated by Transformers and parallelizable recurrent neural networks (RNNs) such as state-space models, yet learning long-term dependencies remains challenging, and state-of-the-art designs trade power consumption for performance.

By Julien Brandoit, Arthur Fyon, Damien Ernst, Guillaume Drion

llms efficiency benchmarks

arXiv Machine Learning

Jul 31

Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis

arXiv:2508. 03105v3 Announce Type: replace Abstract: We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function.

By Yuichi Kondo, Hideaki Iiduka

arXiv AI

Jun 4

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.

By Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Sarthak Mittal, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Ag\"uera y Arcas, Jo\~ao Sacramento

llms benchmarks

arXiv Machine Learning

Jul 20

Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training

arXiv:2607. 15745v1 Announce Type: new Abstract: Common practice when training Convolutional Neural Networks (CNNs) is to use randomly shuffled mini-batches.

By Anxhelo Shehu, Enes Stastoli, Arben Cela

benchmarks

arXiv AI

Jun 15

DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation

arXiv:2506. 14202v4 Announce Type: replace-cross Abstract: End-to-end backpropagation requires storing activations throughout all layers, creating memory bottlenecks that limit model scalability.

By Makoto Shing, Masanori Koyama, Takuya Akiba

llms diffusion

arXiv AI

Jun 30

Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping

arXiv:2606. 29983v1 Announce Type: cross Abstract: Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length algorithmic tasks.

By Hsun-Yu Kuo, El Mahdi Chayti, Patrik Reizinger, Wieland Brendel, Martin Jaggi

llms reinforcement-learning

arXiv AI

Jul 21

Reducing Per-Sample Harm in Stochastic Optimization

arXiv:2607. 16261v1 Announce Type: cross Abstract: Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.

By Apostolos Avranas

computer-vision benchmarks

arXiv AI

6d ago

Reducing Per-Sample Interference in Stochastic Optimization

arXiv:2607. 16261v2 Announce Type: replace-cross Abstract: Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.

By Apostolos Avranas

computer-vision benchmarks

arXiv AI

Jul 21

LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Models

arXiv:2607. 16339v1 Announce Type: new Abstract: Diffusion-based Large Language Models(DLLMs) enable parallel generation via Semi-Autoregressive (SAR) decoding in text generation.

By Xingru Chen, Zelang Liang, Yongjia Ma, Jiqing Zhan, Shuling Yang, Lian Wen, Kun Zhan

llms rag diffusion efficiency

arXiv Machine Learning

Jul 14

Controllably Efficient Language Models

arXiv:2511. 05313v2 Announce Type: replace Abstract: The substantial inference costs of attention in transformers motivated the development of efficient sequence mixers: namely sparse and sliding window attention, convolutions and linear attention.

By Jatin Prakash, Aahlad Puli, Rajesh Ranganath

llms efficiency benchmarks

Related stories