LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high-dimensional full-precision embeddings impose substantial storage and bandwidth overhead on large-scale indexes. In this paper, we present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage.
FastE is a training‑free, plug‑and‑play method that compresses token prefixes in large language model (LLM) embedding inference. It uses a shared fixed threshold on batch‑mean readout‑prefix alignment to decide when to compress and ranks prefix states by readout attention scores to keep the most important ones. Experiments on Qwen3‑Embedding models show that FastE can cut decoder‑backbone FLOPs by over 40% while preserving more than 99% of the original ranking quality across multiple benchmarks and tasks.
By Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie, Lidan Shou, Weiqiang Wang, Gang Chen
The paper introduces a lossy compressive text autoencoder that learns a compressed latent representation of text by applying residual downscaling and upscaling along the time axis, culminating in a low‑dimensional discrete bottleneck. The authors evaluate various quantization methods, training objectives, and datasets, measuring reconstruction quality with BLEU and LLM‑based semantic judgments, and also test downstream performance on question‑answering and semantic similarity tasks. Their results show that the compressed representations achieve compression rates comparable to lossless algorithms (2.24 bits per byte on web text) while maintaining strong reconstruction and downstream task performance.
By Vinko Sabol\v{c}ec, Angelos Katharopoulos, David Grangier
arXiv:2610.29424v1 Announce Type: new
Abstract: Among existing embedding compression approaches, low-rank factorization provides a simple and mathematically principled solution by approximating the e...
By Oliver Giudice, Anna Russo Russo
The paper introduces K-Token Merging, a latent-space compression method that merges each contiguous block of K token embeddings into a single embedding using a lightweight encoder. The compressed sequence is then processed by a LoRA-adapted large language model, while generation continues in the original vocabulary. Experiments on tasks such as structural reasoning, sentiment classification, and code editing demonstrate that K-Token Merging achieves up to 75% input length reduction with minimal performance loss, placing it on the Pareto frontier of performance versus compression.
By Zihao Xu, John Harvill, Ziwei Fan, Yizhou Sun, Hao Ding, Hao Wang
arXiv:2607. 23507v1 Announce Type: cross Abstract: Choosing the right text embedding model is one of the most consequential -- and most frequently under-examined -- decisions in building a retrieval or search system, yet the model that tops a leaderboard is rarely the best choice for a given deployment.
By Madhav S Baidya