HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding
Read the original on arXiv Machine Learning →HyCo is a hybrid drafting method that assigns depth to a server model’s multi‑token prediction head for execution on a mobile NPU, while alternative branches are handled by a small language model in a single pass. By returning only the hidden state, input embedding, and key‑value entries, HyCo reduces server GPU time per token by 38.6% on Spec‑Bench and 29.2% on 32K‑token summarization, and achieves faster delivery and higher savings than an SLM‑only drafter. The approach also incorporates quantization, a restricted vocabulary, a 1K‑token window, and an ε‑threshold rule to relax exact sampling distribution preservation.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.