RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding

RheoSampling tackles the one‑hot dilemma in stochastic dynamic‑tree speculative decoding by decoupling token sampling from tree construction and verification. It injects a sampled token into deterministic top‑K slots, assigning distinct proxy probabilities for expansion and pruning while preserving the true sampling probability for verification, thereby achieving lossless, context‑aware top‑K construction with stochastic sampling. Experiments on various LLMs show higher acceptance rates and speedups compared to existing dynamic‑tree methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• llms
• efficiency
• benchmarks