← Back to all news
arXiv Computation and Language By Collin Zhang, Tingwei Zhang, Vitaly Shmatikov

Speculative Probing: LLM Monitoring at Speculative-Decoding Cost

Read the original on arXiv Computation and Language →

The paper introduces Speculative Probing, a method that repurposes the speculative‑decoding module of large language models for real‑time classification tasks. By appending a trained soft prompt to the target sequence, the approach leverages the already‑cached KV store during inference, adding negligible overhead while achieving higher accuracy than traditional hidden‑state probes. Experiments on four classification tasks across multiple models show that these lightweight probes outperform zero‑shot GPT‑5.4‑mini and rival or surpass specialized 8B safety classifiers without running a full LLM.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Aug 4

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.

By Li Wang, Yi Su, Xiabao Wu, Chiran You, Yongchao Liu, Zhan Qiu, Juelu Zhang, Jiajun Zheng, Fangxin Liu, Jie Zhang, Chen Tian, Chengying Huan
llmsagentsefficiency
More like this →
arXiv Computation and Language
3d ago

Verification-Aware Training for Speculative Decoding

Verification-Aware Training (VAT) is a plug‑in framework that improves speculative decoding for large language models by simulating verification during training and using the resulting accept/reject patterns as supervision. VAT adds a lightweight binary verification head to predict whether each draft token will survive sequential verification, and replaces the fixed per‑position weighting with a verification‑adaptive schedule that keeps full weight up to the first rejection point. When applied to EAGLE‑3 and DFlash on Qwen3‑4B, Qwen3‑8B, and LLaMA‑3.1‑8B, VAT increases average acceptance length by up to 11.4% and wall‑clock speedup by up to 8.7%, yielding consistent gains across math, code, and chat benchmarks.

By Geonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang, Sangmin Lee, Sangdoo Yun, Dongyoon Han
llmsbenchmarks
More like this →