arXiv Computation and Language

Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models

The paper proposes a self‑supervised framework that trains language models to predict concepts—sets of semantically equivalent tokens—rather than single tokens. This approach improves alignment with human similarity judgments, boosts performance on classification, clustering, and reranking tasks, and yields comparable or stronger downstream reasoning while lowering perplexity on semantically meaningful words and only slightly increasing overall perplexity.

arXiv Machine Learning

Sep 1

Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees

The paper investigates why token prediction, a common pre‑training objective for language models, yields useful representations. It introduces a statistical framework linking token prediction accuracy to the geometry of token embeddings, showing that accurate predictions organize embeddings according to Hellinger distances between context distributions. The authors also propose a self‑consistency principle that refines contextual representations through repeated application of a shared block, and provide downstream guarantees for token generation, community recovery, and linear classification.

By Shulei Wang

llms rag

arXiv AI

Jun 6

Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

arXiv:2606. 05173v1 Announce Type: cross Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure.

By Aimen Boukhari

rag diffusion benchmarks

arXiv AI

Aug 21

When Machines Speak: A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models

arXiv:2608. 19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language.

By Su Yan, Rakesh Iyer

llms rag

arXiv Computation and Language

Sep 15

From Token Probabilities to Semantic Constraints: Towards Declarative Probabilistic Evaluation of Language Models

arXiv:2609.13520v1 Announce Type: new Abstract: While Large Language Models have improved rapidly, many fundamental questions remain about how to evaluate the knowledge and reasoning abilities they a...

By Kyle Richardson, Cullen Anderson, Pranav Balakrishnan, Takuto Ban, Daksha Ladia, Ankita Gupta, Marisa Hudspeth

llms

arXiv AI

Jul 23

Sentence Splitter: Uncovering Latent Factual Structure for Self-Supervised Learning

arXiv:2607. 19845v1 Announce Type: cross Abstract: This paper introduces Sentence Splitter, a self-supervised framework built upon a T5-based encoder--decoder architecture for uncovering the latent factual structure of natural language sentences.

By Ahmad Pouramini, Mahsa Afsharizadeh

computer-vision nlp

arXiv AI

Sep 30

ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models

arXiv:2609.36952v1 Announce Type: cross Abstract: Large language models (LLMs) excel at token-level generation but may learn undesirable abstract semantics and lack comprehensive perception. LLM-JEPA...

By Jingnan Pu, Zi-En Fan, Feng Lian

llms rag safety

arXiv AI

Sep 15

Mimir: Large-scale Multilingual Concept Modeling

arXiv:2605.25263v2 Announce Type: replace-cross Abstract: Current language modeling approaches are built around tokens. Text corpora are split into tokens, and models are trained by performing comput...

By Elio Musacchio, Lucia Siciliani, Pierpaolo Basile

llms

arXiv Machine Learning

Sep 22

Guiding the coarse levels of semantic IDs makes the fine levels learnable

arXiv:2609.22227v1 Announce Type: cross Abstract: Generative retrieval represents each item by a short Semantic ID and casts recommendation as autoregressive generation of that sequence. Because the...

By Bin Wang, Zhengyu Zhang

llms rag nlp reinforcement-learning safety

arXiv AI

Sep 28

Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning

The paper introduces TokenAdapt, a model‑agnostic tokenizer transplantation method that uses a hybrid heuristic to initialize new token embeddings, and a novel pre‑tokenization learning approach for multi‑word Supertokens to improve compression. TokenAdapt combines local subword decomposition and global semantic similarity to preserve semantics while reducing retraining needs. Empirical results show that TokenAdapt outperforms existing baselines such as Transtokenizer and ReTok, achieving lower perplexity ratios and significant compression gains.

By Shaurya Sharthak, Vinayak Pahalwan, Adithya Kamath, Adarsh Shirawalmath

llms rag nlp fine-tuning

arXiv Machine Learning

Jun 16

Next-Latent Prediction Transformers Learn Compact World Models

arXiv:2511. 05963v4 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens.

By Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, John Langford

llms diffusion benchmarks safety

arXiv Computation and Language

Sep 1

REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation

arXiv:2608.30627v1 Announce Type: new Abstract: As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token pred...

By Haoran Que, Jiajun Shi, Ting Huang, Renming Pang, Jiaheng Liu, Ge Zhang, Wenhao Huang, Shen Yan, Wei Ye, Shikun Zhang

benchmarks

arXiv AI

Aug 28

Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation

The paper examines how new vocabulary tokens are added to language models for generative recommendation tasks. It shows that the common practice of initializing these tokens as the mean of existing embeddings collapses them into a degenerate subspace, hindering fine‑tuning. The authors propose Grounded Token Initialization (GTI), which places new tokens at semantically meaningful positions in the pretrained embedding space using linguistic supervision, and demonstrate that GTI outperforms mean initialization and other adaptation methods across several benchmarks.

By Daiwei Chen, Zhoutong Fu, Chengming Jiang, Haichao Zhang, Ran Zhou, Tan Wang, Chunnan Yao, Guoyao Li, Rui Cai, Yihan Cao, Ruijie Jiang, Fedor Borisyuk, Jianqiang Shen, Jingwei Wu, Ramya Korlakai Vinayak

llms rag fine-tuning benchmarks

Related stories