The paper investigates how dimensionality reduction and quantization together affect text embedding compression across four task families in the Massive Text Embedding Benchmark using five pretrained models. It finds that combining these techniques can preserve near-original performance while drastically reducing storage, with some classification models retaining at least 99% of their score using less than 0.1% of the original storage. The optimal trade‑off between dimensionality and bit‑width varies by task, with retrieval tasks being more sensitive to compression.
By Riku Kisako, Hayato Tsukagoshi, Ryohei Sasano
The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.
By Yutong Liu, Cairong Zhao, Guosheng Hu
arXiv:2505.10202v2 Announce Type: replace
Abstract: Large Language Models (LLMs) have achieved remarkable success but face significant computational and memory challenges, particularly due to their e...
By Jintian Shao, Hongyi Huang, Jiayi Wu, YiMing Cheng, ZhiYu Wu, You Shan, MingKai Zheng
LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding inference, while high-dimensional full-precision embeddings impose substantial storage and bandwidth overhead on large-scale indexes. In this paper, we present BITEMBED, an extreme low-bit framework for LLM-based text embedding that jointly targets encoding efficiency and vector storage.
FastE is a training‑free, plug‑and‑play method that compresses token prefixes in large language model (LLM) embedding inference. It uses a shared fixed threshold on batch‑mean readout‑prefix alignment to decide when to compress and ranks prefix states by readout attention scores to keep the most important ones. Experiments on Qwen3‑Embedding models show that FastE can cut decoder‑backbone FLOPs by over 40% while preserving more than 99% of the original ranking quality across multiple benchmarks and tasks.
By Jinsong Shu, Jinyong Wen, Baokun Wang, Zhongle Xie, Lidan Shou, Weiqiang Wang, Gang Chen
The paper investigates reducing memory usage of n‑gram embedding tables in open‑weight language models by storing them in low precision. Experiments on nanochat, Llama, and GPT‑2 families show that four‑bit tables cut table memory by about 75% while retaining over 99% of the validation‑loss improvement. The study also demonstrates that re‑allocating the saved memory to larger tables can surpass the performance of full‑precision tables, and that similar savings are achievable in large models like DeepSeek‑V4.1‑Flash and Qwen3.8‑Flash‑Next with minimal impact on perplexity and downstream tasks.
By Stephen Chung, Wenyu Du
arXiv:2510. 04127v2 Announce Type: replace-cross Abstract: Approximate nearest neighbour (ANN) search underpins large-scale retrieval, increasingly within the retrieval-augmented generation pipelines that ground large language models, yet the methods that address it have multiplied across communities until they are seldom read as a single field.
By Sean Moran
arXiv:2605.28190v2 Announce Type: replace
Abstract: Embedding benchmarks like MTEB report a single score per model, implicitly treating robustness as a static, scalar property. We argue that embeddin...
By Manuel Frank, Haithem Afli
arXiv:2610.27699v1 Announce Type: new
Abstract: Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar...
By Xinyu Lei, Hong Huang, Jiebin Zhang, Hongyu Li, Jinlong Song, Decheng Wu, Guanghua Yu, Jianchen Zhu, Sujian Li
arXiv:2606. 04620v1 Announce Type: cross Abstract: LLMs have become the state-of-the-art algorithms for solving NLP tasks.
By Pasindu Wickramasinghe, Achyuta Muthuvelan, Rachmad Vidya Wicaksana Putra, Minghao Shao, Muhammad Shafique
arXiv:2607. 16213v1 Announce Type: new Abstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck.
By Soumia Bouyahiaoui, Manel Kara laouar, Aicha Boutorh, Mohamed Hadj Ameur
The paper introduces the Generative Embedding Benchmark (GEB), which evaluates how much content from an embedding can be recovered by a decoder that only has access to the frozen embedding and a question, without the original image or intermediate features. GEB uses a curated visual‑question‑answering dataset with 1,800 development and 900 test items covering natural images, scene text, and visual documents. Experiments on seven public embedding models show that visual‑only scores range from 28.25 to 33.21, while joint image‑question encoding boosts scores up to 65.56, revealing that generative readout uncovers information bottlenecks not captured by traditional separability‑based benchmarks.
By Yun Li, Biao Yang, Peixi Wu, Yunhao Zhou, Mingzhou Jiang, Wei Yuan, Fan Yang, Wenwu Ou