arXiv AI
Jun 29
MultiHashFormer: Hash-based Generative Language Models
arXiv:2606. 28057v1 Announce Type: cross Abstract: Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size.
Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs
arXiv:2606. 08347v1 Announce Type: cross Abstract: Modern language models represent text using discrete token-level embeddings, which forces recurring multi-token patterns to be learned implicitly across Transformer layers.
Related stories
arXiv AI
Jun 3
Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
arXiv:2606. 03465v1 Announce Type: cross Abstract: Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints.
arXiv Machine Learning
Jul 9
TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models
arXiv:2607. 07388v1 Announce Type: cross Abstract: Large Language Models (LLMs) store factual knowledge and domain-specific patterns implicitly in dense Transformer parameters, making knowledge expansion costly through pretraining, fine-tuning, retrieval augmentation, or longer contexts.
arXiv AI
Jun 11
Augmenting Molecular Language Models with Local $n$-gram Memory
arXiv:2606. 12113v1 Announce Type: cross Abstract: Transformer-based language models for SMILES strings suffer from a locality gap: standard character-level tokenization fragments chemically meaningful motifs, forcing models to repeatedly learn local syntax at the expense of long-range dependencies.
arXiv AI
Jun 2
LLMs Need Encoders for Semantic IDs Too
arXiv:2606. 00324v1 Announce Type: cross Abstract: Multimodal LLMs use dedicated encoders to bridge non-language modalities (vision encoders for images, depth models for audio codec tokens) because raw token embeddings alone cannot capture modality-specific structure.
arXiv Machine Learning
Jun 3
Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning
arXiv:2601. 23169v2 Announce Type: replace Abstract: Current neural architectures lack a principled way to handle interchangeable tokens, i.
Hugging Face Trending Papers
Jun 2
Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures.
arXiv Machine Learning
Jul 14
Tokenizing Numerical and Embedding Features for LLM RecSys
arXiv:2607. 10016v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as backbone architectures for recommender systems because of their strong sequence modeling and representation learning capabilities.
arXiv AI
Jun 10
PromptEmbedder: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting
arXiv:2605. 28066v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable efficacy in text embedding, yet current adaptation methods like LoRA face significant bottlenecks in computational efficiency and cross-architecture transferability.
arXiv AI
Jun 6
Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
arXiv:2606. 05173v1 Announce Type: cross Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure.
arXiv AI
Jul 7
Separating Representation from Reconstruction Enables Scalable Text Encoders
arXiv:2607. 04011v1 Announce Type: cross Abstract: While decoders have rapidly scaled, encoders have remained largely unchanged since BERT.
Hugging Face Trending Papers
Jun 24
BitNet Text Embeddings
LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high-dimensional full-precision embeddings impose substantial storage and bandwidth overhead on large-scale indexes. In this paper, we present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage.