arXiv:2606. 12629v1 Announce Type: cross Abstract: We show that the standard basis of transformer hidden states already provides a training-free, architecture-general feature basis.
By Varun Reddy Nalagatla
StoSignSGD is a new sign‑based optimization algorithm that injects structural stochasticity into the sign operator, ensuring unbiased updates. It resolves the divergence issues of traditional SignSGD on non‑smooth objectives, achieving optimal convergence rates in convex settings and improved complexity bounds in non‑convex, non‑smooth problems. Empirical results show that StoSignSGD is stable and efficient across large language model training, outperforming AdamW and SignSGD in low‑precision regimes (FP8 and FP4) and delivering speedups and accuracy gains on models ranging from OLMo2‑370M to 7B LLMs.
By Dingzhi Yu, Rui Pan, Yuxing Liu, Difan Zou, Tong Zhang
arXiv:2608. 06177v1 Announce Type: new Abstract: Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference.
By Quentin Luquet de Saint-Germain, Massil Ait Abdeslam, Jean Pierre David
arXiv:2603.02069v2 Announce Type: replace
Abstract: We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the pop...
By Jihwan Kim, Dogyoon Song, Chulhee Yun
arXiv:2609.21422v1 Announce Type: cross
Abstract: Deep representation learning often selects hidden features and fits the final predictor on the same sample, so fixed-feature analysis performed after...
By Mahdi Mohammadigohari, Nicole M\"ucke
arXiv:2607. 29674v1 Announce Type: cross Abstract: SignMuon compresses the Muon update to one bit per parameter by taking its elementwise sign, providing the most direct way to run a matrix-aware optimizer under an extremely low communication budget.
By Maria Smirnova, Alexey Kravatskiy