arXiv AI September 16, 2026 By Qihu Xie, Ziwei Li, Yi Kang

FlexEE: Self-Speculative and KV-Compatible Early Exiting for Offloading-Aware LLM Inference

FlexEE is an early‑exiting framework designed for large language model inference that is constrained by computation and memory, particularly in offloading‑based deployments. It uses layer‑wise exit supervision, self‑speculative decoding over a Top‑K local vocabulary, and dynamic hidden‑state management to enable reliable intermediate‑layer predictions and memory‑aware execution. Experiments on Llama2‑7B and Llama3‑8B show that FlexEE achieves significant speedups—up to 1.27×/3.16× and 1.25×/2.83× respectively—while maintaining minimal accuracy loss.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv AI

Jun 16

Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

arXiv:2512. 22420v5 Announce Type: replace-cross Abstract: Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel.

By Rui Li, Zhaoning Zhang, Libo Zhang, Huaimin Wang, Xiang Fu, Zhiquan Lai

llms efficiency benchmarks

Hugging Face Trending Papers

Jul 27

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.

llms efficiency

arXiv Machine Learning

1d ago

SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

arXiv:2609.36590v1 Announce Type: cross Abstract: Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeof...

By Hankun Lin, Patrick Pynadath, Ruqi Zhang

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 13

Accelerating Large Language Model Inference with Self-Supervised Early Exits

arXiv:2407. 21082v3 Announce Type: replace-cross Abstract: This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers.

By Florian Valade

llms benchmarks

arXiv AI

Jul 28

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.

By Dengke Han

llms efficiency

arXiv AI

Jun 11

VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

arXiv:2606. 12243v1 Announce Type: cross Abstract: Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel.

By Yuchen Xian, Yang He, Yunqiu Xu, Yi Yang

llms

One email a morning, machine-written

Related stories

Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

Accelerating Large Language Model Inference with Self-Supervised Early Exits

DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

VIA-SD: Verification via Intra-Model Routing for Speculative Decoding