arXiv Computation and Language
A KAN-based Non-linear Factorization for Embedding Layer Compression
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Aug 28
Squeezing More from Limited Data with Recursive Transformers
The paper investigates how to effectively pre‑train language models when the data budget is limited but compute is plentiful. It shows that increasing model size only improves performance up to an optimal point, after which overfitting degrades generalization, and that this optimal size varies with both the data budget and downstream tasks. To overcome the inefficiencies of standard Transformers in this regime, the authors propose recursive Transformers that reuse a shared block across depth and employ factorized embeddings, achieving better results than standard models on 10M–100M word pre‑training budgets and competitive performance with BabyLM Challenge 2025 winners.
arXiv Computation and Language
Sep 21
VQ-Logits: Compressing the Output Bottleneck of Large Language Models via Vector Quantized Logits
arXiv:2505.10202v2 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved remarkable success but face significant computational and memory challenges, particularly due to their e...
arXiv AI
Sep 10
FastE: Readout-Triggered Token Compression for LLM Embedding Inference
FastE is a training‑free, plug‑and‑play method that compresses token prefixes in large language model (LLM) embedding inference. It uses a shared fixed threshold on batch‑mean readout‑prefix alignment to decide when to compress and ranks prefix states by readout attention scores to keep the most important ones. Experiments on Qwen3‑Embedding models show that FastE can cut decoder‑backbone FLOPs by over 40% while preserving more than 99% of the original ranking quality across multiple benchmarks and tasks.
arXiv AI
Jul 21
LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models
arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.
arXiv Machine Learning
Jun 19
Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings
arXiv:2104. 08928v4 Announce Type: replace-cross Abstract: Unstructured text provides decision-makers with a rich data source in many domains, ranging from product reviews in retail to nursing notes in healthcare.
arXiv Computation and Language
3d ago
When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression
The paper investigates how dimensionality reduction and quantization together affect text embedding compression across four task families in the Massive Text Embedding Benchmark using five pretrained models. It finds that combining these techniques can preserve near-original performance while drastically reducing storage, with some classification models retaining at least 99% of their score using less than 0.1% of the original storage. The optimal trade‑off between dimensionality and bit‑width varies by task, with retrieval tasks being more sensitive to compression.