Accelerate StarCoder with 🤗 Optimum Intel on Xeon: Q8/Q4 and Speculative Decoding

Speculative Decoding on CPUs: Nearly 4x Faster Token Generation with DFlash

Speculative decoding leverages idle CPU resources to accelerate token generation without altering model outputs. In vLLM benchmarks, DFlash achieved a 3.92× increase in autoregressive throughput using Qwen3.5‑9B on an Intel Xeon 6 at a concurrency of 1. The article details the origins of this speedup, discusses acceptance metrics, and outlines factors that influence when speculation is beneficial.

Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

arXiv:2602. 07223v2 Announce Type: replace Abstract: Long-context large language model (LLM) inference has become the norm for today's AI applications.

StarCoder2 and The Stack v2

A Sim-to-Real Study of Surface-Code Decoder Benchmarking

The study benchmarks six quantum error‑correction decoders on the Willow processor, the first device operating below the surface‑code threshold, using a hierarchy of increasingly realistic noise models. By evaluating real hardware data across multiple code distances, bases, and round counts, the authors find that rank agreement with hardware emerges only when each operation type is assigned its own error rate. They also independently test NVIDIA’s Ising pre‑decoder, showing it offers no accuracy‑latency advantage over other decoders in most evaluations, and release the full evaluation pipeline and data for future comparisons.

STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

arXiv:2606. 08382v1 Announce Type: cross Abstract: Low-rank projection has emerged as a promising approach for compressing the KV cache by exploiting hidden-dimension redundancy.

Speculative Decoding for 2x Faster Whisper Inference