arXiv AI
Investigating Model Compression for Neural Machine Translation in the Biomedical Domain
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Oct 27
Improving the Performance of Compressed Transformers with Post-Hoc Embedding Alignment
arXiv:2602.14635v2 Announce Type: replace-cross Abstract: Model compression is necessary for deployment of transformer models in resource-constrained environments. However, compression reduces the pe...
arXiv Computation and Language
Sep 15
Studying quantization trade-offs for efficient inference deployment in machine translation
arXiv:2607.29397v3 Announce Type: replace Abstract: Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low l...
arXiv Machine Learning
5d ago
Trainable Vector Quantization for Large Language Models
The paper introduces Trainable Vector Quantization (TVQ), a method that uses straight-through estimation to allow gradient descent optimization of vector-quantized large language models. TVQ reformulates VQ with trainable latent weights, enabling arbitrary objective minimization and is applied in a block-wise knowledge distillation framework to balance quality and training efficiency. Experiments show TVQ achieves state‑of‑the‑art low‑bit post‑training quantization, achieving near‑lossless 2‑bit quantization for LLaMA2‑70B with only a 1% accuracy drop on question‑answering benchmarks.
arXiv AI
Jul 20
KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
arXiv:2603. 01875v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is an essential technique to compress large language models (LLMs) into smaller ones.
arXiv Machine Learning
Oct 22
A Comprehensive Evaluation on Quantization Techniques for Large Language Models
The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.
arXiv Machine Learning
Sep 2
Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation
arXiv:2602.09691v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is a technique to compress a larger teacher system into a smaller student. In machine translation, KD is commonly...