The paper examines how new vocabulary tokens are added to language models for generative recommendation tasks. It shows that the common practice of initializing these tokens as the mean of existing embeddings collapses them into a degenerate subspace, hindering fine‑tuning. The authors propose Grounded Token Initialization (GTI), which places new tokens at semantically meaningful positions in the pretrained embedding space using linguistic supervision, and demonstrate that GTI outperforms mean initialization and other adaptation methods across several benchmarks.
By Daiwei Chen, Zhoutong Fu, Chengming Jiang, Haichao Zhang, Ran Zhou, Tan Wang, Chunnan Yao, Guoyao Li, Rui Cai, Yihan Cao, Ruijie Jiang, Fedor Borisyuk, Jianqiang Shen, Jingwei Wu, Ramya Korlakai Vinayak
arXiv:2602. 17907v2 Announce Type: replace-cross Abstract: Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity.
By Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini
arXiv:2602. 17907v3 Announce Type: replace-cross Abstract: Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity.
By Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini
arXiv:2510. 04567v3 Announce Type: replace-cross Abstract: Graph Neural Networks (GNNs) are powerful tools for processing relational data but often struggle to generalize to unseen graphs, giving rise to the development of Graph Foundational Models (GFMs).
By Weishuo Ma, Yanbo Wang, Xiyuan Wang, Lei Zou, Muhan Zhang
arXiv:2608. 00542v1 Announce Type: new Abstract: Graphs model relational data throughout science and industry, from citation networks to product co-purchase graphs.
By Zhuoyi Peng, Yi Yang
The paper introduces PromptGFM, a Graph Foundation Model designed for text‑attributed graphs (TAGs). It integrates Large Language Models (LLMs) and Graph Neural Networks (GNNs) through a Graph Understanding Module that prompts LLMs to emulate GNN workflows, and a Graph Inference Module that creates a language‑based graph vocabulary for better alignment and scalability. Experiments show PromptGFM outperforms existing methods and transfers effectively across various graphs and tasks.
By Xi Zhu, Haochen Xue, Ziwei Zhao, Wujiang Xu, Jingyuan Huang, Minghao Guo, Qifan Wang, Kaixiong Zhou, Imran Razzak, Yongfeng Zhang
The paper investigates why token prediction, a common pre‑training objective for language models, yields useful representations. It introduces a statistical framework linking token prediction accuracy to the geometry of token embeddings, showing that accurate predictions organize embeddings according to Hellinger distances between context distributions. The authors also propose a self‑consistency principle that refines contextual representations through repeated application of a shared block, and provide downstream guarantees for token generation, community recovery, and linear classification.
By Shulei Wang
arXiv:2606. 14047v1 Announce Type: cross Abstract: Long-context language modeling requires not only extending context windows but maintaining coherent understanding of entity states and relationships across thousands of tokens -- a challenge that semantic similarity alone cannot address.
By Ghadir Alselwi, Basem Suleiman, Hao Xue, Shoaib Jameel, Hakim Hacid, Flora D. Salim, Imran Razzak
arXiv:2606. 29773v1 Announce Type: new Abstract: Graphs are widely used to model relational systems, with applications in domains such as social networks, finance, and biomedicine.
By Haoxin Sun, Yiqing Lin, Yajun Huang, Chenhui Dong, Mingjun Li, Zhongzhi Zhang
arXiv:2604. 25853v3 Announce Type: replace-cross Abstract: Traditional loss functions, including cross-entropy, contrastive, triplet, and su pervised contrastive losses, used for fine-tuning pre-trained language models such as BERT, operate only within local neighborhoods and fail to account for the global semantic structure.
By Aditya Sharma, Vinti Agarwal, Rajesh Kumar
arXiv:2606. 11898v1 Announce Type: cross Abstract: Research on Text-Attributed Graphs (TAGs) has gained significant attention recently due to its broad applications across various real-world data scenarios, such as citation networks, e-commerce platforms, social media, and web pages.
By Hengyi Feng, Zeang Sheng, Meiyi Qiang, Meiyi Qiang, Wentao Zhang
The paper introduces TokenAdapt, a model‑agnostic tokenizer transplantation method that uses a hybrid heuristic to initialize new token embeddings, and a novel pre‑tokenization learning approach for multi‑word Supertokens to improve compression. TokenAdapt combines local subword decomposition and global semantic similarity to preserve semantics while reducing retraining needs. Empirical results show that TokenAdapt outperforms existing baselines such as Transtokenizer and ReTok, achieving lower perplexity ratios and significant compression gains.
By Shaurya Sharthak, Vinayak Pahalwan, Adithya Kamath, Adarsh Shirawalmath