arXiv Machine Learning

AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing

AIR-LLM is an edge inference architecture that broadcasts large language model (LLM) weights over radio, allowing edge devices to perform matrix-vector multiplications directly in the RF domain without storing or loading the weights. The system uses MIMO spatial multiplexing and an energy‑efficient precoder‑postcoder pair to reduce airtime and calibrate the wireless channel, enabling a single broadcast to serve unlimited users. Experiments on real urban channel models show that AIR-LLM achieves only a 4.0% perplexity loss on LLaMA‑3.1‑8B while saving energy by up to 157.7× compared to FP16 and reducing airtime by over 100× for 20 users.

arXiv AI

Sep 11

Improving 5G AI-RAN MCS Selection by Predicting Retransmissions

The paper introduces NOSTRAdAMUS, a predictive link‑adaptation framework for 5G NR that forecasts retransmissions in the next radio frame using recent HARQ history and adjusts the Modulation and Coding Scheme accordingly. Gradient Boosting models achieve 82.9% overall accuracy, with high‑confidence predictions correct 94.2% of the time and a 5.5 µs inference latency. Evaluated OTA on the X5G testbed and various channel emulators, the approach boosts goodput by up to 71.5% and cuts retransmissions by up to 71.8% without retraining across diverse scenarios.

By Tamerlan Aghayev, Maxime Elkael, Michele Polese, Reshma Prasad, Salvatore D'Oro, Yunseong Lee, Koichiro Furueda, Tommaso Melodia

efficiency benchmarks

arXiv AI

Jul 7

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

arXiv:2504. 08791v3 Announce Type: replace-cross Abstract: On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low throughput and capability.

By Zonghang Li, Tao Li, Wenjiao Feng, Rongxing Xiao, Jianshu She, Hong Huang, Mohsen Guizani, Hongfang Yu, Qirong Ho, Wei Xiang, Xue Liu

llms efficiency safety

arXiv AI

Jul 8

Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

arXiv:2607. 05475v1 Announce Type: cross Abstract: Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency.

By Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, Jiliang Wang

llms efficiency safety

arXiv Machine Learning

Jul 17

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

arXiv:2607. 14618v1 Announce Type: new Abstract: CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs.

By Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam, Sanggeon Yun, Ryozo Masukawa, Mohsen Imani

llms efficiency

arXiv Machine Learning

Jun 11

NARRAS: Edge-Triggered Distributed Inference for CSI-Based Localization in Vehicular IoT Networks

arXiv:2606. 11914v1 Announce Type: cross Abstract: CSI-based localization with spatially distributed antenna arrays exposes a basic resource trade-off.

By Rodrigo Oliver, Ricardo Vazquez Alvarez, Alejandro Lancho, Stefano Rini

arXiv Machine Learning

Sep 10

Non-Coherent Over-the-Air Federated Learning: Protocol, Convergence, and Device Scheduling

arXiv:2609.08312v1 Announce Type: cross Abstract: To mitigate the scalability bottleneck in the radio access network (RAN) in federated edge learning (FEEL), over-the-air federated learning (AirFL) e...

By Haifeng Wen, Nicol\`o Michelusi, Osvaldo Simeone, Yang Yang, Hong Xing

safety

arXiv AI

Aug 7

BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

arXiv:2608. 05926v1 Announce Type: cross Abstract: Edge inference is a promising paradigm to provide large language model (LLM) inference services in next-generation mobile networks.

By Guanqiao Qu, Shuo Chen, Qian Chen, Kin K. Leung, Xianhao Chen

llms efficiency

arXiv Machine Learning

Sep 21

TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching

arXiv:2609.21172v1 Announce Type: new Abstract: Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications...

By Zhihao Shu, Md Musfiqur Rahman Sanim, Jie Hu, Kun Yuan, Minghai Qin, Gagan Agrawal, Wei Niu

llms efficiency

arXiv Machine Learning

Sep 23

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.

By Liam Cooper, Shinnung Jeong, Hyeran Jeon, Jeffrey Young, Hyesoon Kim

llms benchmarks

arXiv Machine Learning

Sep 4

LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

LeanStream introduces a speculate‑and‑refine streaming framework that enables efficient on‑device inference of large language models by progressively refining computation, loading, and cache‑retention priorities using partial GPU results. This approach allows fine‑grained overlap between GPU execution and storage I/O, avoiding the trade‑offs of existing systems that serialize execution or incur redundant weight fetches. Implemented on mobile and embedded platforms, LeanStream reduces memory usage by 4.8× to 7.5× compared to prior work while improving token generation throughput by 1.6× to 2.1×.

By Renyuan Liu (Richard), Yuyang Leng (Richard), Kaiyan Liu (Richard), Yuzhou Zhong (Richard), Shaohan Hu (Richard), Chun-Fu (Richard), Chen, Peijun Zhao, Heechul Yun, Shuochao Yao

llms efficiency safety

arXiv Computation and Language

Sep 10

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

arXiv:2609.09662v1 Announce Type: cross Abstract: Deploying Large Language Models (LLMs) directly on mobile platforms at the edge is gaining traction due to a myriad of benefits, such as increased pr...

By Weisi Yang, Stephen Xia

llms benchmarks safety

arXiv AI

4d ago

OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streaming

arXiv:2609.34653v2 Announce Type: replace Abstract: On-device streaming omni-modal inference safeguards user privacy and eliminates prohibitive per-token API costs, but faces a critical bottleneck: t...

By Zongshang Shen, Wangsong Yin, Daliang Xu, Mengwei Xu, Xuanzhe Liu

llms efficiency multimodal benchmarks safety

Related stories