Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

6,536 stories · RSS feed

arXiv AI
Jul 2

Diffusion-GR2: Diffusion Generative Reasoning Re-ranker

arXiv:2607. 01170v1 Announce Type: cross Abstract: Generative reasoning re-rankers achieve strong recommendation accuracy by emitting a chain-of-thought before re-ordering a candidate list, but they are slow at inference: an autoregressive (AR) decoder spends one sequential forward pass per reasoning token, and the reasoning trace far exceeds the ranking it produces.

By Zhuoxuan Zhang (Yang), Kangqi Ni (Yang), Yuhang Chen (Yang), Mingfu Liang (Yang), Xiaohan Wei (Yang), Yunchen Pu (Yang), Fei Tian (Yang), Chonglin Sun (Yang), Frank Shyu (Yang), Adam (Yang), Song, Sandeep Pandey, Luke Simon, Tianlong Chen, Xi Liu
arXiv Machine Learning
Jul 2

Efficient Compression of Structured and Unstructured Volumes via Learned 3D Gaussian Representation

arXiv:2607. 01164v1 Announce Type: new Abstract: Recent work has shown that implicit neural representations (INRs) can be trained to effectively compress structured and unstructured volume data, allowing for direct data querying with a reduced memory footprint.

By Landon Dyken, Sharmistha Chakrabarti, Nathan Debardeleben, Steve Petruzza, Qi Wu, Will Usher, Sidharth Kumar
arXiv AI
Jul 2

FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices

arXiv:2604. 25421v2 Announce Type: replace-cross Abstract: Federated fine-tuning provides a practical route to adapt large language models (LLMs) on edge devices without centralizing private data, yet in mobile deployments the training wall-clock is often bottlenecked by straggler-limited uplink communication under heterogeneous bandwidth and intermittent participation.

By Changyu Li, Shuanghong Huang, Jiashen Liu, Ming Lei, Jidu Xing, Kaishun Wu, Lu Wang, Fei Luo
arXiv Machine Learning
Jul 2

ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces

arXiv:2607. 01125v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance.

By Xun Dong, Yibo Xu, Naigang Wang, Xin Li, Penghang Yin, Zi Yang
arXiv AI
Jul 2

ConRTF: Edge-Constrained Boundary Distribution Refinement for Realtime TransFormer Table Structure Recognition

arXiv:2607. 00734v1 Announce Type: cross Abstract: Table Structure Recognition (TSR) aims to recover the row and column layout of tables from document images, a key step in document understanding pipelines.

By Eliott Thomas, Tri-Cong Pham, Mickael Coustaty, Aurelie Joseph, Gaspar Deloin, Vincent Poulain d'Andecy, Jean-Marc Ogier, Antoine Doucet
arXiv Machine Learning
Jul 2

Leveraging Multimodality for Real-Time Classification of Transients and Variables found by the Zwicky Transient Facility

arXiv:2607. 00228v1 Announce Type: cross Abstract: Modern time-domain surveys such as the Zwicky Transient Facility (ZTF) generate hundreds of thousands of alerts each night, making real-time decisions for follow-up observations a central challenge in time-domain astronomy.

By Ved G. Shah, Nabeel Rehemtulla, Adam A. Miller, Sushant Sharma Chaudhary, Michael W. Coughlin, Antoine Le Calloch, Matthew J. Graham, Joahan Castaneda Jaimes, Theophile Jegou du Laz, Ashish A. Mahabal, Frank J. Masci, Josiah Purdum, Reed Riddle, Jesper Sollerman, Anastasia Wei, Mansi M. Kasliwal