Hugging Face Trending Papers June 25, 2026

Structure Before Collapse: Transient semantic geometry in next-token prediction

Read the original on Hugging Face Trending Papers →

Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs. This creates a puzzle: next-token prediction language models are trained predominantly (as context length increases) with one-hot labels: the same context is very unlikely to appear twice in training with different labels.

Summary generated by The Flow from the publisher's feed. The full article lives at Hugging Face Trending Papers.

arXiv Machine Learning

Jun 26

Structure Before Collapse: Transient semantic geometry in next-token prediction

arXiv:2606. 26749v1 Announce Type: new Abstract: Neural Collapse predicts that balanced one-hot classification pushes model representations to be equally far from each other; a symmetric configuration that depends only on the output label and ignores any semantic similarity in the inputs.

By Yize Zhao, Isabel Papadimitriou, Christos Thrampoulidis

llms rag efficiency

arXiv Machine Learning

Jun 16

Emergent retokenization symmetry in large language models: phenomenology and applications

arXiv:2606. 15521v1 Announce Type: cross Abstract: Tokenization introduces representational redundancy: under a fixed token vocabulary, every byte string admits many valid token encodings, or segmentations, that decode to the same surface string.

By Kanishk Jain, Matthew Day, Tankut Can

llms computer-vision nlp benchmarks

arXiv Machine Learning

Jun 18

Decomposing Prediction Mechanisms for In-Context Recall

arXiv:2507. 01414v2 Announce Type: replace Abstract: We introduce a new family of toy problems that combine features of linear-regression-style continuous in-context learning (ICL) with discrete associative recall.

By Sultan Daniels, Dylan Davis, Dhruv Gautam, Wentinn Liao, Gireeja Ranade, Anant Sahai

llms efficiency

arXiv AI

Jul 21

Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysis

arXiv:2606. 07559v2 Announce Type: replace-cross Abstract: Fine-tuning a language model often fails silently when its correct completion must outrank a near-synonym competitor.

By Vaibhav Prakash, Jayasri Dontabhaktuni

llms rag fine-tuning

arXiv Machine Learning

Jun 2

A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization

arXiv:2606. 00230v1 Announce Type: new Abstract: Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs.

By Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande, Anna Rumshisky

llms

arXiv AI

Jun 6

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

arXiv:2606. 05173v1 Announce Type: cross Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure.

By Aimen Boukhari

rag diffusion benchmarks

Related stories

Structure Before Collapse: Transient semantic geometry in next-token prediction

Emergent retokenization symmetry in large language models: phenomenology and applications

Decomposing Prediction Mechanisms for In-Context Recall

Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysis

A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning