arXiv Computation and Language August 31, 2026 By Collin Zhang, Tingwei Zhang, Vitaly Shmatikov

Speculative Probing: LLM Monitoring at Speculative-Decoding Cost

Read the original on arXiv Computation and Language →

The paper introduces Speculative Probing, a method that repurposes the speculative‑decoding module of large language models for real‑time classification tasks. By appending a trained soft prompt to the target sequence, the approach leverages the already‑cached KV store during inference, adding negligible overhead while achieving higher accuracy than traditional hidden‑state probes. Experiments on four classification tasks across multiple models show that these lightweight probes outperform zero‑shot GPT‑5.4‑mini and rival or surpass specialized 8B safety classifiers without running a full LLM.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Aug 4

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.

By Li Wang, Yi Su, Xiabao Wu, Chiran You, Yongchao Liu, Zhan Qiu, Juelu Zhang, Jiajun Zheng, Fangxin Liu, Jie Zhang, Chen Tian, Chengying Huan

llms agents efficiency

arXiv AI

Jun 10

Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs

arXiv:2606. 10487v1 Announce Type: cross Abstract: Deploying large language models in user-facing systems requires efficient output safety filtering.

By Huizhen Shu, Xuying Li, Piao Xue

llms safety

arXiv AI

Jul 7

SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

arXiv:2607. 03333v1 Announce Type: cross Abstract: LLM agents are becoming a common interface for research, coding, and question answering, yet their Thought-Action-Observation loop is often serial: the model reasons, emits a tool call, then idles the GPU until the result returns.

By Huajun Bai, Weiwei Lv, Huichuan Zheng, Youyou Lu, Jiwu Shu

llms agents nlp benchmarks

arXiv Machine Learning

Sep 30

SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

arXiv:2609.36590v1 Announce Type: cross Abstract: Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeof...

By Hankun Lin, Patrick Pynadath, Ruqi Zhang

llms fine-tuning benchmarks

arXiv Computation and Language

Sep 4

Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding

The paper introduces AdaptiveSpec, a training‑free speculative decoding method that simultaneously adapts the per‑step verification rule and the draft‑tree shape using signals generated during decoding. It replaces the fixed token‑match rule with a margin‑based threshold and adjusts tree depth, width, and node count based on draft confidence and recent acceptance history, allowing the total draft count to vary. Experiments on SGLang show up to 56% throughput gains over EAGLE‑3 while maintaining 93% of lossless task accuracy on GSM8K, MATH‑500, and HumanEval across three models.

By Oszk\'ar Urb\'an, Young D. Kwon, Stylianos I. Venieris, Cecilia Mascolo

llms efficiency benchmarks

arXiv AI

Jun 16

Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

arXiv:2512. 22420v5 Announce Type: replace-cross Abstract: Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel.

By Rui Li, Zhaoning Zhang, Libo Zhang, Huaimin Wang, Xiang Fu, Zhiquan Lai

llms efficiency benchmarks

One email a morning, machine-written

Related stories

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs

SPORK: Self-Speculative Forking to Accelerate Agentic LLM Inference

SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding

Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving