arXiv:2606. 02765v1 Announce Type: cross Abstract: Model dimension ($d_{model}$) is a fundamental hyperparameter in transformer language models, yet its role in setting the geometric limits of feature representation remains under-explored.
By Alexander Guha
arXiv:2506. 08774v2 Announce Type: replace-cross Abstract: Different machine learning models can represent the same underlying concept in different ways.
By Fan Xu, Luis A. Leiva
The paper extends mechanistic interpretability of large language models by modeling concepts as low‑dimensional non‑linear manifolds rather than linear subspaces. It introduces a concept‑based alignment (CBA) score to compare these manifolds across layers and models, revealing block structures in intermediate layers, a shift from syntax‑dominated to mixed syntactic‑semantic concepts, and training‑dependent multilingual sharing. The study also shows that alignment patterns differ across model families and training stages, with adjacent stages aligning more closely than distant ones.
By Tido Specht, Elias Benedict Krey, Nils Neukirch, Nils Strodthoff
The paper introduces a flow matching framework that unifies various neural representational dissimilarity metrics under a single theoretical umbrella. By interpreting these metrics as Jeffreys divergences with different velocity constraints, the authors demonstrate that flow matching improves distance estimation for complex distributions and continuous variables. The framework also facilitates the principled design of new dissimilarity measures.
By Zeyuan Ye, Xue-Xin Wei
arXiv:2608. 15215v1 Announce Type: cross Abstract: Token-level knowledge distillation (KD) matches two conditional distributions per position, yet the standard objectives compare them pointwise: a Kullback-Leibler gradient is blind to which wrong token receives probability mass.
By Gordei Verbii, Juho Lee
arXiv:2607. 09405v1 Announce Type: new Abstract: Similarity search is a primary application of embedding models trained by contrastive learning.
By Nick Whiteley