arXiv Computation and Language
HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding
Read the original on arXiv Computation and Language →HyFlash is a compact parallel drafter designed for on-device speculative decoding. It introduces a prefix‑memory prior that separates long‑range context modeling from fine‑grained draft‑position interactions, using local attention for recent context and a recurrent memory to compress historical target features into fixed‑size states. Experiments demonstrate that HyFlash achieves higher acceptance rates than full‑context attention while requiring substantially less KV‑cache storage, especially benefiting shallow drafters and long contexts.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.