The paper identifies two numerical bugs in standard softmax, log‑softmax, and cross‑entropy implementations: Softmax Collapse, where a dominant logit causes its gradient to be rounded to zero, and Softmax Cancellation, where student and teacher logits that are equal still produce non‑zero gradients. The authors provide straightforward implementations that correct both issues. Their fix removes the previously observed "Slingshot" phenomenon, accelerates grokking with weight decay, prevents accuracy collapse without weight decay, keeps student networks aligned with teachers during distillation, and demonstrates that post‑memorization logit growth is bounded by an equilibrium between weight decay and Adam’s epsilon hyperparameter.
By Jonathan T. Barron, Dan Gnanapragasam, Michael Tschannen
arXiv:2606. 00262v1 Announce Type: cross Abstract: InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected.
By Melihcan Erol, Suat Evren, Oktay Ozel, Alexander Morgan, Jongha Jon Ryu, Lizhong Zheng
arXiv:2607. 22258v1 Announce Type: new Abstract: Deep learning models using traditional softmax classifiers have achieved remarkable success in various classification tasks.
By Yi-Hang Zhu, Rajeev Raman, Shiqi Su, Jianyuan Sun, Xinyu Yang, Nan Xing, Huiyu Zhou
The paper addresses the challenge of creating machine learning learners that can guarantee provably correct predictions in difficult test-time scenarios, such as adversarial attacks and natural distribution shifts. It introduces a reliable learner with optimal theoretical guarantees for these settings and discusses practical implementations. The authors demonstrate strong performance on examples like linear separators under log-concave distributions and smooth boundary classifiers under smooth probability distributions.
By Maria-Florina Balcan, Steve Hanneke, Rattana Pukdee, Dravyansh Sharma
The paper argues that empirical results in Machine Learning are often difficult to reproduce due to limited availability of code and supporting materials, which hampers research progress. It analyzes and quantifies these challenges and proposes concrete measures to enhance the verifiability of results, even if full reproducibility cannot be guaranteed. The authors provide their code and supporting resources on GitHub for reference.
By Samet Hicsonmez, Nermin Samet, Renaud Marlet
arXiv:2610. 34934v1 Announce Type: new Abstract: Machine learning systems trained on untrusted data are vulnerable to data and backdoor poisoning attacks, in which an adversary corrupts training and test samples to manipulate predictions at inference time.
By Evan Rose, Alina Oprea
The paper investigates how gating the value pathway in attention mechanisms provides two missing capabilities of softmax attention: abstention and noise filtering. Experiments on models ranging from 10M to 350M parameters show that abstention benefits smaller models while noise filtering becomes more advantageous as models scale, and that combining both primitives yields the best performance across all sizes. The authors also demonstrate that the gates effectively suppress interference and that each gate type has a distinct blind spot, all while adding negligible parameters and preserving compatibility with key‑value caching.
By Richard Zhe Wang
arXiv:2606. 01746v1 Announce Type: cross Abstract: Modern neural networks are highly susceptible to adversarial perturbations.
By Kai Wang
arXiv:2610.17588v1 Announce Type: cross
Abstract: Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full mo...
By Aditya Somasundaram, Charles Guille-Escuret, Alexander Moreno, Zhengzhong Liu, Eric Xing
arXiv:2608.27704v1 Announce Type: new
Abstract: When machine learning classifiers are retrained, inputs correctly classified by the previous model version may be misclassified by the updated version,...
By Madhusudan Srinivasan, Namith Nishal Raphae
The paper introduces a diagnostic test for class sensitivity in saliency methods, revealing that many popular techniques produce nearly identical explanations regardless of the predicted class. This limitation appears across different architectures and datasets, indicating a structural issue. To address this, the authors propose CASE, a contrastive explanation method that isolates features uniquely discriminative for the predicted class, and demonstrate its improved fidelity and class specificity through experiments.
By Dane Williamson, Yangfeng Ji, Matthew Dwyer
Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearity that follows. This...