Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

3,248 stories · RSS feed

Hugging Face Trending Papers
Jul 19

OrderMoE: An expert similarity driven distributed edge MoE inference

Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inference over resource-constrained and bandwidth-limited edge infrastructures. Existing distributed MoE serving methods mainly rely on exact expert placement, caching, replication, or communication scheduling, while overlooking the functional similarity among experts, which provides an opportunity to reduce cross-server token transmission.

arXiv Machine Learning
Jul 17

Real-Time Detection of Charge Jumps in Superconducting Qubits with a Convolutional Neural Network

arXiv:2607. 14293v1 Announce Type: cross Abstract: Ionizing radiation from cosmic rays and gammas can induce discontinuous jumps in the environmental charge of superconducting qubits (charge jumps), causing correlated errors that challenge fault-tolerant quantum computing while simultaneously providing a detection signature for quantum sensing applications.

By Daniel Gaytan-Villarreal, Peter Meiring, Daniel Baxter, Daniel Bowring, Grace Bratrud, Matteo Cremonesi, Giuseppe Di Guglielmo, Grace Wagner, Bowen Xiao
arXiv Machine Learning
Jul 17

Low-Latency Relay Selection in NR-V2X Vehicular Communications via Graph Isomorphism Networks with Edge Features

arXiv:2607. 14176v1 Announce Type: new Abstract: Reliable, low-latency uplink connectivity is a key requirement for C-V2X networks in dense urban environments, where fast channel variations and blockages often degrade direct vehicle-to-infrastructure links.

By Giambattista Amati, Federica Mangiatordi, Emiliano Pallotti, Simone Angelini, Pierpaolo Salvo, Paola Vocca
arXiv AI
Jul 17

Multibit neural inference in a N-ary crossbar architecture

arXiv:2604. 26979v2 Announce Type: replace-cross Abstract: In-memory computing (IMC) is a paradigm that enables neural network inference by computing analog matrix-vector multiplications (MVM) directly in memory crossbar arrays, with the potential for energy efficiency gains over conventional von Neumann architectures.

By Anatole Moureaux, Anthony Lopes Temporao, Flavio Abreu Araujo
arXiv Machine Learning
Jul 17

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

arXiv:2607. 14618v1 Announce Type: new Abstract: CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs.

By Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam, Sanggeon Yun, Ryozo Masukawa, Mohsen Imani