The paper investigates loss spikes that occur late in language model pretraining, attributing them to delayed, terminal divergences in output logits—termed time‑bomb spikes. It shows that these spikes are preceded by rapidly growing quantization error in the LM head, and that reducing numerical precision speeds up the spikes while increasing precision slows them. The authors propose replacing AdamW with SGD with momentum on the LM head to suppress logit growth, and find that various mitigation techniques reduce quantization error, indicating a common numerical cause for stability.
By Zaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum
arXiv:2606. 05516v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distributed across layers.
By Wanhao Yu, Ziyan Wang, Zheng Wang, Abeer Matar Almalky, Yihang Zuo, Shuteng Niu, Sen Lin, Adnan Siraj Rakin, Deliang Fan, Li Yang
arXiv:2608.23358v1 Announce Type: new
Abstract: The performance gap between low- and high-resource languages in LLMs is widely known, but it remains unclear which internal model factors drive these d...
By Francois Meyer, Jan Buys
The paper proposes Embedding Space Separation (ES2), a fine‑tuning technique that increases the distance between harmful and harmless embeddings in large language models. ES2 uses two regularization losses—a embedding anchor loss and a KL‑divergence loss—to preserve the model’s general capabilities while enhancing safety. Experiments show that ES2 effectively defends against various attack methods with only modest performance loss on some tasks.
By Xu Zhao, Xiting Wang, Weiran Shen
The study demonstrates that large language models can be compressed by clustering weight matrices into only 16–64 shared values without retraining, preserving strong accuracy. Fine‑tuning the cluster centroids recovers most of the remaining accuracy loss. Analysis reveals that maintaining the relative rank of weights and their overall variance is crucial for performance, while early layers and spatial geometry of cluster assignments also play significant roles.
By Borja Aizpurua, Sukhbinder Singh, Rom\'an Or\'us
arXiv:2502. 11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining.
By Guoxia Wang, Shuai Li, Congliang Chen, Jinle Zeng, Jiabin Yang, Dianhai Yu, Yanjun Ma, Li Shen