The paper introduces a new weight update method for morphological neural networks based on the Multiplicative Weights Update scheme. It treats each morphological perceptron as a learning-from-experts problem in logarithmic space, using a correlation-based reward that favors inputs aligned with the desired output change even when gradient signals are weak. Experiments on fully connected layers and transformer components across nine benchmarks show that correlational training improves performance on eight tasks, up to 32.84 percentage points, and reduces run‑to‑run variability.
arXiv:2505. 09710v4 Announce Type: replace Abstract: We investigate deep morphological neural networks (DMNNs), studying how changes in algebraic structure affect the expressivity and trainability of deep architectures.
By Konstantinos Fotopoulos, Petros Maragos
arXiv:2607. 09967v1 Announce Type: cross Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps.
By Ethan Smith
arXiv:2602. 19799v2 Announce Type: replace-cross Abstract: Despite recent algorithmic advances, we still lack principled ways to leverage the well-documented rescaling symmetries in ReLU neural network parameters.
By Arthur Lebeurrier, Titouan Vayer, R\'emi Gribonval
arXiv:2609.07086v1 Announce Type: new
Abstract: Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their su...
By Hemanth Saratchandran, Simon Lucey
arXiv:2610.20337v1 Announce Type: new
Abstract: The success of transformers is attributed to several factors, including their capacity to express formal languages over variable-length inputs, and abi...
By Yaomengxi Han, Debarghya Ghoshdastidar
arXiv:2606. 25971v1 Announce Type: new Abstract: Modern neural network training relies on optimizers such as Adam and Muon which act on each weight matrix as a single object.
By Alexander H\"agele, Alejandro Hern\'andez-Cano, Atli Kosson, Martin Jaggi
The paper introduces Cubit, a Transformer‑style architecture that replaces the standard attention mechanism with Kernel Ridge Regression (KRR). By interpreting attention as Nadaraya‑Watson regression, Cubit incorporates the closed‑form KRR solution, combining kernel‑based value aggregation with normalization via the inverse kernel matrix. The authors also propose a Limited‑Range Rescale (LRR) to stabilize training and report that Cubit shows improved long‑sequence modeling, with gains increasing as training sequence length grows.
By Chuanyang Zheng, Jiankai Sun, Yihang Gao, Yuehao Wang, Liangchen Tan, Mac Schwager, Anderson Schneider, Yuriy Nevmyvaka, Xiaodong Liu
arXiv:2410. 24050v3 Announce Type: replace Abstract: Large-scale pretraining of transformers has been central to the success of foundation models.
By Ambroise Odonnat, Wassim Bouaziz, Vivien Cabannes
The paper proposes a memory‑efficient training method for transformers by training a local proxy network for each block to predict the global output via distillation. This allows gradients to be propagated locally while closely approximating the true backpropagation gradients, achieving a better accuracy‑memory trade‑off on CIFAR‑100 vision transformers. The approach also improves fine‑tuning of large language models under tight memory budgets, outperforming other memory‑efficient zeroth‑order algorithms.
By Wu S. Zihan, Brian A. Yuan, Guillaume Bellec
arXiv:2605. 15435v2 Announce Type: replace Abstract: Standard deep-learning pipelines usually choose the network architecture before training and keep it fixed throughout optimization.
By Lute Lillo, Nick Cheney