arXiv:2606. 26617v1 Announce Type: new Abstract: Scaling laws describe how learning performance varies with model size, data size, and compute.
arXiv:2602. 24012v2 Announce Type: replace Abstract: Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models.
arXiv:2605. 24316v3 Announce Type: replace Abstract: Mini-batching is central to large-scale optimization, yet its role in statistical scaling laws remains limited.
arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.
The paper investigates the often-overlooked scale vectors in large language models, showing that despite their tiny size they are crucial for pre‑training performance. The authors provide theoretical insights that scale vectors mainly aid optimization rather than expressivity, and they analyze how weight decay affects different normalization layers. Building on these findings, they propose lightweight improvements—branch‑specific heterogeneity, better placement, and magnitude‑direction reparameterization—that consistently reduce loss across a range of model sizes and training settings.
arXiv:2605. 07914v2 Announce Type: replace Abstract: Sharpness-aware and gradient-alignment methods have been shown to improve generalization, however each family of methods targets a single geometric property of the loss landscape, while ignoring the other.
arXiv:2402.11215v4 Announce Type: replace Abstract: The choice of batch size in minibatch stochastic gradient optimization is critical for both optimization and generalization performance in large-sc...
arXiv:2606. 18650v1 Announce Type: new Abstract: As Large Language Model (LLM) datasets scale to trillions of tokens, data selection has emerged as a critical frontier to filter out uninformative noise and construct adaptive learning trajectories.
arXiv:2607. 12916v1 Announce Type: new Abstract: In this work, we introduce CoCo, a loss function aimed at learning normalized and well-structured representations.
arXiv:2607. 17673v1 Announce Type: cross Abstract: Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs.
The paper investigates the expressive power of multimodal contrastive learning architectures by treating them as parameterized families of joint density estimators. It shows that the classic two‑tower CLIP model is a universal approximator for two modalities, while a common extension that sums pairwise similarities fails to approximate arbitrary joint distributions when three or more modalities are involved, though it can match all pairwise conditionals. To address this limitation, the authors introduce Hadamard‑CLIP, which adds a single learned weight vector to restore universal approximation for any number of modalities while retaining CLIP’s efficient retrieval capabilities.
arXiv:2509. 24882v2 Announce Type: replace Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models.
arXiv:2608. 09523v1 Announce Type: new Abstract: Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success.