arXiv:2602. 17063v2 Announce Type: replace-cross Abstract: Sub-bit model compression targets storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck.
arXiv:2606. 12629v1 Announce Type: cross Abstract: We show that the standard basis of transformer hidden states already provides a training-free, architecture-general feature basis.
StoSignSGD is a new sign‑based optimization algorithm that injects structural stochasticity into the sign operator, ensuring unbiased updates. It resolves the divergence issues of traditional SignSGD on non‑smooth objectives, achieving optimal convergence rates in convex settings and improved complexity bounds in non‑convex, non‑smooth problems. Empirical results show that StoSignSGD is stable and efficient across large language model training, outperforming AdamW and SignSGD in low‑precision regimes (FP8 and FP4) and delivering speedups and accuracy gains on models ranging from OLMo2‑370M to 7B LLMs.
arXiv:2608. 06177v1 Announce Type: new Abstract: Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference.
arXiv:2603.02069v2 Announce Type: replace Abstract: We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the pop...
arXiv:2609.21422v1 Announce Type: cross Abstract: Deep representation learning often selects hidden features and fits the final predictor on the same sample, so fixed-feature analysis performed after...
arXiv:2607. 29674v1 Announce Type: cross Abstract: SignMuon compresses the Muon update to one bit per parameter by taking its elementwise sign, providing the most direct way to run a matrix-aware optimizer under an extremely low communication budget.
arXiv:2607. 08779v1 Announce Type: cross Abstract: The signed integer alphabet contains one more negative representable value than positive.
arXiv:2606. 20076v1 Announce Type: cross Abstract: Latent Diffusion Models (LDMs) have become dominant in visual synthesis, but their quality-compute trade-off is largely constrained by the tokenizer's fixed compression ratio.
LionMuon is a new optimizer that alternates between Lion’s sign-based updates and Muon’s spectral matrix-sign updates on a fixed period P, sharing a single dual-EMA momentum buffer. This design keeps the memory footprint the same as Lion and half that of AdamW while reducing the average iteration cost compared to Muon. Experiments on 124M, 355M, and 720M models show LionMuon Pareto-dominates Muon, Lion, Signum, and AdamW across datasets and architectures, achieving lower validation loss with less compute.
SignSeek is a new method for learning transferable sign representations that enables efficient retrieval of signs from dictionaries using only a query video. It employs contrastive learning with saliency‑guided articulator masking, aligning same‑gloss signs across signers while focusing on the single most critical articulator per sign. Trained on 266K samples from multiple sign languages, SignSeek achieves state‑of‑the‑art cross‑corpus retrieval performance and zero‑shot generalisation to unseen British Sign Language, also improving isolated sign recognition and subtitle alignment.
arXiv:2602. 02355v2 Announce Type: replace-cross Abstract: Hierarchical federated learning (HFL) is well suited for large-scale wireless and Internet of Things systems, where devices communicate with nearby edge servers before reaching the cloud.
arXiv:2607. 08754v1 Announce Type: cross Abstract: Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss.