Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

6,536 stories · RSS feed

arXiv Machine Learning
Jul 3

Q-GAIN: A Python Package for Machine Learning and Physically Informed Analysis Applications

arXiv:2607. 02413v1 Announce Type: cross Abstract: Here we describe the quantum gas analysis and inference (Q-GAIN) Python package, which enables rapid deployment of machine learning (ML) and physics-informed analysis techniques for cold-atom experiments.

By M. Doris, S. Guo, S. M. Koh, L. Ritter, A. R. Fritsch, S. Mukherjee, I. B. Spielman, J. P. Zwolak
arXiv AI
Jul 3

Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

arXiv:2607. 01480v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal.

By Ye Liu, Srijan Bansal, Bo Pang, Yang Li, Zeyu Leo Liu, Yifei Ming, Zixuan Ke, Shafiq Joty, Semih Yavuz
arXiv AI
Jul 3

DemoPSD: Disagreement-Modulated Policy Self-Distillation

arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.

By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
arXiv AI
Jul 3

MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering

arXiv:2607. 01420v1 Announce Type: cross Abstract: As grounded QA systems are increasingly deployed in AI assistants, accurately attributing generated answers to evidence is critical for user trust and model safety.

By Dang Quang Thien Tran, Quang V. Dang, Vinamra Tyagi, Sai Soorya Rao Veeravalli, Trang Nguyen, Ryan A. Rossi, Franck Dernoncourt, Nedim Lipka, Koustava Goswami, Samyadeep Basu
arXiv Machine Learning
Jul 3

Population-Scale Segmentation of Penile Tissue in DIXON MRI using Deep Learning for Quantitative Phenotyping in Male Reproductive Health

arXiv:2607. 02127v1 Announce Type: cross Abstract: Penile measurement is clinically relevant across male reproductive and urogenital health, including conditions such as micropenis, congenital and endocrine disorders, and sexual or urinary dysfunction.

By Jan Ernsting, Gunnar Paul Kordes, Nils Johannaber, Lynn Ogoniak, Wolfgang Roll, Tim Hahn, Alexander Siegfried Busch, Benjamin Risse
arXiv AI
Jul 3

Low-Latency Task-Oriented Image Transmission with Opportunistic Spectrum Access

arXiv:2607. 01921v1 Announce Type: cross Abstract: Communication systems designed for reliable data reconstruction, rather than task-oriented communication, typically rely on separate source and channel coding and incur high latency under limited spectrum availability and fading channels.

By Jo\~ao Henrique Inacio de Souza, Mattia Merluzzi, Mateus P. Mota, Beatriz Soret, Petar Popovski
arXiv Machine Learning
Jul 3

SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication

arXiv:2607. 01678v1 Announce Type: new Abstract: Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale.

By Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang
arXiv AI
Jul 3

Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generation

arXiv:2607. 01590v1 Announce Type: new Abstract: Developing high-performance kernels for Neural Processing Units (NPUs) is a critical industry bottleneck, requiring developers to manually navigate implicit hardware constraints and strict memory hierarchies.

By Junyi Wen, Ruiyan Zhuang, Yongjia Xu, Pengtu Li, Rui Zou, Hongyi Chen, Chingman Wan, Puxu Yang, Wuhui Chen, Yanlin Wang
arXiv Machine Learning
Jul 3

WBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution

arXiv:2607. 02097v1 Announce Type: cross Abstract: Large kernel depthwise convolutions achieve strong performance but suffer from significant degradation as kernel size grows due to irregular memory access from gather-based computation; while Large Kernel Acceleration (LKA) helps on small feature maps, it becomes counterproductive on large feature maps, even slower than non-accelerated implementations.

By Wan Song, Wei Zhou, Rui Wang, Jun Yu, Toru Kurihara, Jiajia Xu, Shu Zhan