Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

3,411 stories · RSS feed

Hugging Face Trending Papers
Jul 2

Online Segment 3D Gaussians via Launching Virtual Drones

Interactive segmentation of 3D Gaussians offers a compelling opportunity for real-time manipulation of 3D scenes, thanks to the real-time rendering capability of 3D Gaussian Splatting (3DGS). However, existing methods require a time-consuming per-scene setup - typically tens of seconds or even minutes - before interactive segmentation can begin on a raw 3DGS scene.

arXiv Machine Learning
Jul 1

Rethinking Continual Anomaly Detection on the Edge: Benchmarking Under Realistic Industrial Conditions

arXiv:2605. 24251v2 Announce Type: replace Abstract: Continual anomaly detection (CAD) addresses the need for industrial inspection systems to adapt to evolving production conditions, yet existing methods share three critical gaps: unrealistic evaluation, no systematic comparison, and no consideration of edge deployment constraints.

By Chad Weatherly, Sen Lin
arXiv Machine Learning
Jul 1

Zero-Shot Quantization for Object Detectors using Off-the-Shelf Generative Models

arXiv:2606. 31456v1 Announce Type: new Abstract: With an increasing number of Object Detection (OD) models being deployed on edge devices, Zero-Shot Quantization for OD (ZSQ-OD) aims to quantize these models when access to the original training data is prohibited.

By Hyunho Lee, Kyomin Hwang, Hyeonjin Kim, Suyoung Kim, Sunghyun Wee, Nojun Kwak
arXiv Machine Learning
Jul 1

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.

By Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du