Hugging Face Trending Papers
Property-Based Testing for Robust Softmax and Cross-Entropy
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
Oct 23
A Fix for Softmax and Cross-Entropy
The paper identifies two numerical bugs in standard softmax, log‑softmax, and cross‑entropy implementations: Softmax Collapse, where a dominant logit causes its gradient to be rounded to zero, and Softmax Cancellation, where student and teacher logits that are equal still produce non‑zero gradients. The authors provide straightforward implementations that correct both issues. Their fix removes the previously observed "Slingshot" phenomenon, accelerates grokking with weight decay, prevents accuracy collapse without weight decay, keeps student networks aligned with teachers during distillation, and demonstrates that post‑memorization logit growth is bounded by an equilibrium between weight decay and Adam’s epsilon hyperparameter.
arXiv AI
Jun 2
When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE
arXiv:2606. 00262v1 Announce Type: cross Abstract: InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected.
arXiv Machine Learning
Jul 27
Class-Balanced Softmax: A Bayes Theory-Based Method for Long-Tailed Recognition
arXiv:2607. 22258v1 Announce Type: new Abstract: Deep learning models using traditional softmax classifiers have achieved remarkable success in various classification tasks.
arXiv Machine Learning
Sep 17
Reliable learning in challenging environments
The paper addresses the challenge of creating machine learning learners that can guarantee provably correct predictions in difficult test-time scenarios, such as adversarial attacks and natural distribution shifts. It introduces a reliable learner with optimal theoretical guarantees for these settings and discusses practical implementations. The authors demonstrate strong performance on examples like linear separators under log-concave distributions and smooth boundary classifiers under smooth probability distributions.
arXiv AI
Sep 30
Position: Let's Strengthen Verifiability If We Can't Enforce Reproducibility
The paper argues that empirical results in Machine Learning are often difficult to reproduce due to limited availability of code and supporting materials, which hampers research progress. It analyzes and quantifies these challenges and proposes concrete measures to enhance the verifiability of results, even if full reproducibility cannot be guaranteed. The authors provide their code and supporting resources on GitHub for reference.
arXiv Machine Learning
5d ago
A Certified Robustness Framework for Backdoor Poisoning Attacks using $f$-Divergences
arXiv:2610. 34934v1 Announce Type: new Abstract: Machine learning systems trained on untrusted data are vulnerable to data and backdoor poisoning attacks, in which an adversary corrupts training and test samples to manipulate predictions at inference time.