← Back to all news
arXiv AI

Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data

The paper investigates whether supervised contrastive learning objectives produce a regular simplex geometry in representation space when trained on long‑tailed data. It shows that existing objectives fail to achieve this geometry because they depend on class sample counts. The authors propose a fully balanced contrastive objective that provably forms a regular simplex, leading to a more balanced representation space and improved performance on long‑tailed recognition and transfer learning tasks.

Related stories

arXiv AI
Oct 26

Internal Discrepancy for Contrastive Learning: Generalization Bounds and Downstream Behaviors

The paper introduces the concept of internal discrepancy, measuring the gap between the mean representation of positive samples and latent cluster centers in contrastive learning. It derives a generalization bound for downstream supervised tasks that depends on this discrepancy, providing a unifying framework for existing assumptions and motivating a relaxed k‑means loss to reduce it. The study also explores how contrastive learning affects downstream behavior, particularly in class‑imbalanced settings, and establishes properties of the resulting supervised losses.

By Jialiang Wang, Qing Ma, Xiong Zhou, Gangfeng Hu, Haoliang Li, Xianming Liu
More like this →
arXiv Machine Learning
Sep 11

Divergence-Based Similarity Function for Multi-View Contrastive Learning

The paper introduces a divergence-based similarity function (DSF) for multi-view contrastive learning, representing each set of augmented views as a distribution and measuring similarity via distribution divergence. DSF captures joint structure across all views, outperforming prior pairwise methods on tasks such as kNN classification, linear evaluation, transfer learning, and distribution shift. It also offers greater efficiency and eliminates the need for a temperature hyperparameter, unlike cosine similarity.

By Jaehyoung Jeon, Cheolsu Lim, Myungjoo Kang
fine-tuning
More like this →
arXiv Machine Learning
Jun 16

InfoNCE Induces Gaussian Distribution

arXiv:2602. 24012v2 Announce Type: replace Abstract: Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models.

By Roy Betser, Eyal Gofer, Meir Yossef Levi, Guy Gilboa
safety
More like this →
arXiv Machine Learning
Aug 19

Expressivity In Multimodal Contrastive Learning

The paper investigates the expressive power of multimodal contrastive learning architectures by treating them as parameterized families of joint density estimators. It shows that the classic two‑tower CLIP model is a universal approximator for two modalities, while a common extension that sums pairwise similarities fails to approximate arbitrary joint distributions when three or more modalities are involved, though it can match all pairwise conditionals. To address this limitation, the authors introduce Hadamard‑CLIP, which adds a single learned weight vector to restore universal approximation for any number of modalities while retaining CLIP’s efficient retrieval capabilities.

By Andrew Stuart, Florian Wolf
llmsragdiffusionmultimodal
More like this →