← Back to all news
arXiv AI By Prateek Kumar Sikdar, Arpan Ghosh

A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives

Read the original on arXiv AI →

The paper presents a rigor‑matched audit comparing two periodic‑step, search‑based layer‑skipping methods for efficient large language model inference: a confidence‑gated early‑exit baseline (ConfLayers) and a self‑speculative decoding approach (SWIFT). Across two Qwen2.5 model scales and tasks (GSM8K reasoning and CNN/DailyMail summarization), SWIFT consistently outperforms ConfLayers in accuracy and, after separating search overhead, achieves faster true inference speed in most settings. The study also evaluates two trained‑routing methods (LayerRoute and LayerDrop), finding modest speedups but significantly lower accuracy, especially for LayerRoute on GSM8K at 1.5B.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 15

LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference

LayerRoute is a parameter‑efficient technique that enables adaptive skipping of transformer layers in large language models. It adds a lightweight per‑layer router (~21.5K parameters) and LoRA adapters (rank 8, ~1.08M parameters) to each of the 24 blocks in Qwen2.5‑0.5B‑Instruct, training them jointly with a gate‑regularized language‑modeling objective. Across ten independent runs, the method consistently identifies nine middle layers as skip‑eligible, achieves a verified wall‑clock speedup of 1.02x–1.06x, and improves perplexity by an average of 1.16 points, while the router’s decisions vary per input, confirming genuine adaptive behavior.

By Prateek Kumar Sikdar
llmsfine-tuningefficiency
More like this →
arXiv AI
Oct 7

Evaluating Escalation Signals for LLM Routing: Targets, Controls, and Five Ways to Fool Yourself

The paper investigates whether semantic entropy—a measure of disagreement among a language model’s sampled answers—can serve as a cheap signal for deciding when to route a query from a small to a larger language model. Experiments on GSM8K and other benchmarks show that semantic entropy can distinguish small‑model mistakes and improve routed accuracy, but the authors also reveal that a simple question‑difficulty rule can mimic its performance and that other factors (definition of success, benchmark design, live sampling cost) can undermine its effectiveness. They propose a checklist of checks to validate escalation signals and demonstrate how to predict when a cached‑outcome approach will fail. "whyItMatters":"The study highlights the importance of rigorous evaluation of escalation signals, showing that seemingly promising metrics can be misleading without proper controls and that practical routing decisions must account for cost and benchmark design."

By Ramin Pishehvar, Andrea Morandi, Mahesh Viswanathan
llmsbenchmarkssafety
More like this →
arXiv AI
Oct 31

Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models

The paper investigates why speculative decoding (SD) performs better than expected on sparse mixture-of-experts (MoE) models. By evaluating 10 large MoE models, the authors find that recent fine‑grained models achieve higher speedups at low batch sizes, contrary to prior predictions of a concave speedup‑batch relationship. They attribute this to lower verification costs driven by temporal overlap of draft tokens and GPU execution dynamics, and develop a maximum‑entropy IPF framework that accurately predicts unique expert counts from routing overlap, closely matching real‑model behavior.

By Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli
More like this →