The paper introduces an attack that reconstructs text generated by locally hosted large language models by monitoring CPU cache activity during detokenization. It uses Flush+Reload on shared tokenizer code to time decoding, then Prime+Probe to capture token‑dependent cache traces, followed by a clustering‑and‑language‑model pipeline to recover the output text. The method is evaluated across various datasets, hardware, inference frameworks, and model families, successfully retrieving semantically accurate outputs from real‑world local LLM deployments, including agentic systems.
By Roy Weiss, Benyamin Konstantinov, Eitam Sheetrit, Tomer Simon, Yisroel Mirsky
arXiv:2512. 22420v5 Announce Type: replace-cross Abstract: Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel.
By Rui Li, Zhaoning Zhang, Libo Zhang, Huaimin Wang, Xiang Fu, Zhiquan Lai
arXiv:2607. 01313v1 Announce Type: cross Abstract: In practice, most commercial LLM providers do not publicly release details of underlying LLM architectures.
By Christopher Ellis, Shreyas Chaudhari, Mei-Yu Wang, Leighton Barnes, Giulia Fanti, Jos\'e M. F. Moura
SpecGuard is an inference‑time backdoor detector that leverages speculative decoding—using a small draft model to propose tokens and a target model to verify them—without adding extra model computation. By monitoring the draft‑token acceptance rate, SpecGuard detects when a target model shifts toward attacker‑controlled behavior while the draft model does not, signaling a backdoor trigger. The method reliably identifies a range of backdoor types, including stealthy attacks that bypass input‑level filters, across multiple model families.
By Rui Wen, Ahmed Salem, Andrew Paverd, Mark Russinovich, Zheng Li
Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.
arXiv:2609.38830v1 Announce Type: new
Abstract: Sparse attention is widely used to accelerate long-context inference in modern large language models (LLMs), but its input-dependent execution behavior...
By Fahao Chen, Linkang Du, Jinhao Zhou, Peng Li, Zhou Su
SpecGuard is an inference‑time backdoor detector that leverages speculative decoding—using a draft model to propose tokens and a target model to verify them—without adding extra model computation. By monitoring the draft‑token acceptance rate, SpecGuard identifies when a target model’s behavior shifts toward attacker‑controlled outputs, a signal that appears whenever a backdoor is triggered. The method works across various backdoor types and model families, reliably detecting stealthy attacks that bypass input‑level filters while avoiding the extra generation cost of existing runtime detectors.
arXiv:2606. 17104v1 Announce Type: cross Abstract: As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge.
By Shun Usami, Venkatram Vishwanath, E. Wes Bethel
arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.
By Dengke Han
arXiv:2607. 05147v1 Announce Type: new Abstract: Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification.
By Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang
arXiv:2607. 17283v1 Announce Type: new Abstract: Single-stream autoregressive decoding of large language models is bound by memory bandwidth: each generated token requires one full forward pass through the target model, and successive passes cannot be parallelized.
By Param Chordiya
arXiv:2609.12551v2 Announce Type: replace-cross
Abstract: AI is beginning to make substantive contributions to LLM inference optimization. Existing AI optimizations are predominantly profiling-based....
By Ziyue Yang, Yuting Jiang, Lei Qu, Peng Cheng