Hugging Face Blog
Accelerate StarCoder with 🤗 Optimum Intel on Xeon: Q8/Q4 and Speculative Decoding
Read the original on Hugging Face Blog →The Flow has not summarised this story yet — read it at Hugging Face Blog.
Related stories
arXiv Machine Learning
Jun 2
Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention
arXiv:2602. 07223v2 Announce Type: replace Abstract: Long-context large language model (LLM) inference has become the norm for today's AI applications.
Hugging Face Blog
Feb 28, 2024
StarCoder2 and The Stack v2
arXiv AI
Jun 9
STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control
arXiv:2606. 08382v1 Announce Type: cross Abstract: Low-rank projection has emerged as a promising approach for compressing the KV cache by exploiting hidden-dimension redundancy.
Hugging Face Blog
Dec 20, 2023
Speculative Decoding for 2x Faster Whisper Inference
arXiv Machine Learning
Jul 7
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
arXiv:2607. 04244v1 Announce Type: new Abstract: This report describes our approach to the Efficient Qwen Competition, where the goal is to enable low-latency serving of Qwen3.
arXiv Machine Learning
Jul 27
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
arXiv:2607. 22389v1 Announce Type: cross Abstract: With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck.