arXiv Machine Learning
Does a Shared Temperature Imply a Shared Angular Scale in Probabilistic Contrastive Learning?
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
2d ago
How Accurate Is Accurate Enough?
arXiv:2609.38785v1 Announce Type: new Abstract: How accurate must a numerical approximation be within a learning system? Primitive error alone cannot answer this question: errors of the same magnitud...
arXiv Machine Learning
Jul 14
A Strong Balanced-Softmax Classifier-Retraining Baseline for Long-Tailed Recognition
arXiv:2607. 09832v1 Announce Type: new Abstract: Long-tailed recognition methods often modify losses, margins, or representations to reduce the dominance of frequent classes.
arXiv Machine Learning
Jun 11
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
arXiv:2606. 12171v1 Announce Type: cross Abstract: Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs.
arXiv Computer Vision
Sep 3
Breaking the Geometric Bottleneck: Contrastive Expansion in Asymmetric Cross-Modal Distillation
The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.
arXiv Machine Learning
Jul 28
The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers
arXiv:2607. 23050v1 Announce Type: new Abstract: Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it?
arXiv AI
Jun 18
Generalized Kullback-Leibler Divergence Loss
arXiv:2503. 08038v2 Announce Type: replace-cross Abstract: In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of (1) a weighted Mean Square Error (wMSE) loss and (2) a Cross-Entropy loss incorporating soft labels.