arXiv:2606. 30625v1 Announce Type: cross Abstract: Contrastive embedding models trained with scale-invariant losses are typically paired with distance metrics like cosine similarity, effectively ignoring embedding magnitudes.
arXiv:2608.30315v1 Announce Type: new Abstract: Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern lang...
arXiv:2609.15152v1 Announce Type: cross Abstract: Multimodal embedding models encode heterogeneous inputs into a shared embedding space, enabling efficient similarity computation across modalities an...
arXiv:2607. 09405v1 Announce Type: new Abstract: Similarity search is a primary application of embedding models trained by contrastive learning.
arXiv:2504. 16318v3 Announce Type: replace Abstract: Cosine similarity is a standard comparison rule for learned representations in information retrieval, natural language processing, computer vision, and multimodal learning.
The paper investigates why token prediction, a common pre‑training objective for language models, yields useful representations. It introduces a statistical framework linking token prediction accuracy to the geometry of token embeddings, showing that accurate predictions organize embeddings according to Hellinger distances between context distributions. The authors also propose a self‑consistency principle that refines contextual representations through repeated application of a shared block, and provide downstream guarantees for token generation, community recovery, and linear classification.
arXiv:2602. 24012v2 Announce Type: replace Abstract: Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models.
arXiv:2607. 08377v1 Announce Type: new Abstract: Uncertainty quantification is central to the reliable deployment of large language models (LLMs), and eigenvalues of semantic embeddings have recently emerged as a key tool in state-of-the-art methods.
arXiv:2609.06663v1 Announce Type: cross Abstract: Although multimodal Large Language Models (MLLMs) excel in diverse tasks, their scalability remains limited by the memory and computational overhead...
The paper presents a mean‑field analysis of attention in language models, defining an average attention kernel that propagates representations layer by layer. When conditioned on a whole corpus, the kernel predicts the average evolution of representation geometry; when conditioned on a single context, it predicts the expected geometry for that context. The difference between actual attention and the mean‑field prediction—called the mean‑field deviation—captures context‑specific computation, revealing how models diverge from average behavior during training and in few‑shot tasks.
arXiv:2607. 17770v1 Announce Type: cross Abstract: Within Explainable Artificial Intelligence, mechanistic interpretability uses Sparse Autoencoders (SAEs) to extract more interpretable features from neural representations.
arXiv:2503.00612v2 Announce Type: replace-cross Abstract: The basic problem of semantic compression is to minimize the length of a message while preserving its meaning. This differs from classical no...
The paper investigates the often-overlooked scale vectors in large language models, showing that despite their tiny size they are crucial for pre‑training performance. The authors provide theoretical insights that scale vectors mainly aid optimization rather than expressivity, and they analyze how weight decay affects different normalization layers. Building on these findings, they propose lightweight improvements—branch‑specific heterogeneity, better placement, and magnitude‑direction reparameterization—that consistently reduce loss across a range of model sizes and training settings.