arXiv AI
Jul 1
Large Databases Need Small, Open-Weight Language Models
arXiv:2606. 31808v1 Announce Type: new Abstract: Language model systems built around proprietary APIs often operate on a token-based cost model.
Apertus LLM Family Expansion via Distillation and Quantization
arXiv:2605. 29128v2 Announce Type: replace Abstract: The wide adoption of LLMs has led to their use in great variety of applications and scenarios, such as chatbot assistants and data annotation, creating the need for the models to satisfy certain budget and hardware constraints.
Related stories
arXiv AI
Jun 16
LLM4RTL: Tool-Assisted LLM for RTL Generation
arXiv:2606. 15500v1 Announce Type: cross Abstract: Large language models (LLMs) have facilitated impressive progress in software engineering, code generation, tooling, and systems.
arXiv AI
Jun 26
CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
arXiv:2606. 26650v1 Announce Type: cross Abstract: In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs.
arXiv Machine Learning
2d ago
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
arXiv:2609.39223v2 Announce Type: new Abstract: Large language model (LLM) inference is increasingly moving toward lower precision to realize the throughput of hardware accelerators, but aggressive p...
arXiv AI
Aug 5
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
arXiv:2608. 02975v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated impressive performance in MQM-based translation quality (TQ) evaluation, and recent advances in large reasoning models (LRMs) promise even greater improvements.
Hugging Face Trending Papers
Jun 25
CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes.
arXiv Machine Learning
Sep 21
SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference
SpecQuant is a training‑free framework that merges speculative decoding with multi‑parent quantization to enable adaptive, efficient inference of large language models. It generates several quantized variants (INT4, FP8, FP16) from a single base model and routes queries to the appropriate variant based on predicted complexity, using lightweight models for simple tasks and full‑precision models for complex reasoning. Evaluations on Qwen2.5 models across MMLU, AlpacaEval, and GSM8K show 35‑43% speedups with less than 2% accuracy loss, facilitating practical on‑device LLM deployment without specialized infrastructure.
arXiv Machine Learning
Sep 4
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
arXiv:2609. 04010v1 Announce Type: new Abstract: Large Language Models (LLMs) owe much of their success to next-token prediction (NTP), but their autoregressive (AR) structure requires slow, sequential token generation.
arXiv AI
Aug 19
Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Task
The paper explores energy-aware knowledge distillation for large language models (LLMs) used in software engineering tasks such as clone detection, vulnerability prediction, and code summarization. It shows that the commonly used FLOPs metric does not reliably reflect actual energy consumption, and that using energy-surrogate models during distillation can reduce inference energy by up to 90% and memory usage by 86% with only modest accuracy loss. The study demonstrates that guiding distillation with direct energy estimates improves the sustainability and deployability of LLMs on consumer hardware.
arXiv Machine Learning
Jul 8
FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models
arXiv:2607. 05711v1 Announce Type: new Abstract: Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications.
arXiv AI
Sep 2
HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference
HBQ: Hierarchical Scaling Block Quantization with Hardware‑Efficiency‑Aware Design for Accurate LLM Inference proposes a new block‑quantization scheme that uses large blocks and low‑overhead significand scaling to balance hardware efficiency and accuracy. The authors demonstrate that larger blocks improve efficiency by amortizing dequantization and accumulation costs, while their SIG scaling compensates for the resulting accuracy loss. Experiments on a 28 nm ASIC accelerator show that HBQ achieves up to 4.6× higher area/energy efficiency than state‑of‑the‑art weight‑only quantization, with 1.5–3.0× speedup and 1.6–3.3× system energy reduction over existing BQ methods.
arXiv Computation and Language
Sep 1
Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
The paper presents a budget‑aware compression pipeline for deploying 70B‑parameter language models on a single NVIDIA GPU. It examines how pruning, quantization, and KV‑cache compression interact, showing that layer‑wise pruning improves weight quantization robustness and that KV‑cache sparsification complements INT8 KV quantization without harming decoding speed. Using these insights, the authors compressed a 70B model to ~33 GB, achieving ~57 tokens/s on 10k‑token prompts on an A40 while maintaining accuracy within 5% on standard benchmarks.