arXiv AI

Speculative Evaluation of Stochastic LLMs

The paper introduces Speculative Evaluation, a method to reduce variance in evaluating stochastic large language models (LLMs) under a fixed rollout budget. It employs a Hierarchical Bayesian Neyman (HBN) policy that first runs a short uniform pilot, then pools task-level success counts via a hierarchical Bayesian model to compute posterior expectations of task-level sampling variances. Using these expectations, the method applies exact positive-integer Neyman allocation to allocate rollouts, and an asynchronous variant (HBN-async) speculatively executes continuations from partial pilot feedback to mitigate synchronization overhead. Across six checkpoints and 18 benchmark groups, Speculative Evaluation achieves 12.8%-33.6% lower variance compared to uniform allocation, outperforming empirical and independent Bayesian baselines, and demonstrates practical benefits in real-generation experiments.

arXiv Computation and Language
Sep 21

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Draft-OPD introduces an on‑policy distillation method for speculative draft models, addressing the mismatch between supervised fine‑tuning and inference by letting the target model supervise the drafter on draft‑induced states. The approach uses target‑assisted rollouts for stable continuations and replays drafting from error positions exposed during verification, enabling the drafter to learn from both accepted and rejected proposals. Experiments demonstrate that Draft‑OPD achieves more than five‑fold lossless acceleration across diverse tasks, outperforming prior draft models such as EAGLE‑3 and DFlash by 23 % and 13 % respectively.

By Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang, Qianjia Cheng, Xiaoye Qu, Ganqu Cui, Bowen Zhou, Ning Ding, Yun Luo, Yu Cheng
arXiv Machine Learning
Sep 30

Tail-Influence Sampling for CVaR Policy Evaluation

arXiv:2609.38096v1 Announce Type: new Abstract: Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can requi...

By Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer, Rasul Tutunov, Haitham Bou-Ammar
arXiv Machine Learning
Oct 8

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

The paper introduces the Expected Decoding Rounds (EDR) objective, a theoretical framework that models speculative decoding as a Markov reward process and directly optimizes the number of decoding rounds. EDR weights local rejection costs by state occupancies, eliminating auxiliary hyperparameters and enabling unbiased stochastic optimization via temporal‑difference gradients. Applying EDR to fine‑tune state‑of‑the‑art draft models (DSpark and DFly) improves mean accepted length and outperforms prior training objectives across nine benchmarks in math reasoning, code generation, and chat.

By Yunxiao Zhao, Changxiao Cai
arXiv Computation and Language
Aug 25

Can Large Language Models "Hyper-Thread"?

arXiv:2608.22376v1 Announce Type: new Abstract: Large language models generate tokens sequentially, but can they execute multiple tasks concurrently while forming each token? Broader attention alloca...

By Fei Ding
arXiv AI
Oct 15

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fi...

By Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, Leowei Liang
arXiv Machine Learning
Oct 28

Multiple Importance Sampling for Guided Speculative Inference

The paper extends Guided Speculative Inference (GSI) for test‑time scaling of large language models by allowing an ensemble of smaller LLMs to propose draft reasoning steps, which are then verified by a larger base model and a reward model. Using multiple importance sampling (MIS), the authors introduce weighting schemes to combine samples from diverse proposal distributions, prove new bounds on KL divergence and expected reward loss, and show that the balance heuristic is optimal. Experiments demonstrate that combining several diverse LLMs, while keeping the total number of samples fixed, achieves competitive downstream performance and is mathematically robust when a single draft model’s support does not cover the target distribution, a scenario where standard GSI may fail.

By Joseph Rowan, Ashish J. Khisti