The paper examines how new vocabulary tokens are added to language models for generative recommendation tasks. It shows that the common practice of initializing these tokens as the mean of existing embeddings collapses them into a degenerate subspace, hindering fine‑tuning. The authors propose Grounded Token Initialization (GTI), which places new tokens at semantically meaningful positions in the pretrained embedding space using linguistic supervision, and demonstrate that GTI outperforms mean initialization and other adaptation methods across several benchmarks.
arXiv:2608. 07816v1 Announce Type: cross Abstract: Recent advances in generative recommendation (GR) leverage large language models (LLMs) as recommender backbones, enabling LLMs to directly generate recommendations conditioned on item-interaction histories.
arXiv:2511.22707v2 Announce Type: replace-cross Abstract: In web environments, user preferences are often refined progressively as users move from browsing broad categories to exploring specific item...
arXiv:2607. 25209v1 Announce Type: cross Abstract: Generative recommendation commonly represents items using fixed-length semantic identifiers (SIDs) constructed through clustering and quantization.
The paper introduces TokenAdapt, a model‑agnostic tokenizer transplantation method that uses a hybrid heuristic to initialize new token embeddings, and a novel pre‑tokenization learning approach for multi‑word Supertokens to improve compression. TokenAdapt combines local subword decomposition and global semantic similarity to preserve semantics while reducing retraining needs. Empirical results show that TokenAdapt outperforms existing baselines such as Transtokenizer and ReTok, achieving lower perplexity ratios and significant compression gains.
The paper investigates why token prediction, a common pre‑training objective for language models, yields useful representations. It introduces a statistical framework linking token prediction accuracy to the geometry of token embeddings, showing that accurate predictions organize embeddings according to Hellinger distances between context distributions. The authors also propose a self‑consistency principle that refines contextual representations through repeated application of a shared block, and provide downstream guarantees for token generation, community recovery, and linear classification.
arXiv:2608. 03494v1 Announce Type: cross Abstract: Vocabulary extension is an efficient way to adapt pretrained large language models (LLMs) to new languages, but the initialization of newly added token embeddings can strongly affect continued pre-training (CPT) efficiency.
arXiv:2608. 19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural language.
arXiv:2608.30315v1 Announce Type: new Abstract: Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern lang...
arXiv:2606. 11023v1 Announce Type: cross Abstract: Sequential recommendation aims to predict users' next interaction with items by analyzing their historical behavior.
arXiv:2608. 16269v1 Announce Type: cross Abstract: Recent advances in neural topic models with pre-trained language models (PLMs) have achieved strong performance by leveraging general-domain pre-training, yet their topic interpretability often degrades on specialized corpora.
Sequential recommendation aims to predict users' next interaction with items by analyzing their historical behavior. However, the limited quality of item representations remains a critical bottleneck.
arXiv:2606. 17276v1 Announce Type: cross Abstract: Generative recommendation (GR) has emerged as a promising direction for recommender systems.