← Back to all news
arXiv Computation and Language By Christine Zhang, Dan Jurafsky, Chen Shani

Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models

Read the original on arXiv Computation and Language →

The paper proposes a self‑supervised framework that trains language models to predict concepts—sets of semantically equivalent tokens—rather than single tokens. This approach improves alignment with human similarity judgments, boosts performance on classification, clustering, and reranking tasks, and yields comparable or stronger downstream reasoning while lowering perplexity on semantically meaningful words and only slightly increasing overall perplexity.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Sep 1

Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees

The paper investigates why token prediction, a common pre‑training objective for language models, yields useful representations. It introduces a statistical framework linking token prediction accuracy to the geometry of token embeddings, showing that accurate predictions organize embeddings according to Hellinger distances between context distributions. The authors also propose a self‑consistency principle that refines contextual representations through repeated application of a shared block, and provide downstream guarantees for token generation, community recovery, and linear classification.

By Shulei Wang
llmsrag
More like this →