Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

6,122 stories · RSS feed

arXiv Machine Learning

Sep 1

Perforated Backpropagation: A Neuroscience Inspired Extension to Artificial Neural Networks

arXiv:2501.18018v2 Announce Type: replace-cross Abstract: The neurons of artificial neural networks were originally invented when much less was known about biological neurons than is known today. Our...

By Rorry Brenner, Laurent Itti

efficiency benchmarks

arXiv Machine Learning

Sep 1

Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache

arXiv:2608.30252v1 Announce Type: new Abstract: Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of token...

By Tong Yuan, Chengxi Liao, Zeyi Wen

llms agents nlp efficiency

arXiv Machine Learning

Sep 1

RSLM: Training-Free Vector Quantization for Approximate Nearest Neighbor Search

arXiv:2608.30384v1 Announce Type: new Abstract: By introducing RSLM (Rotated Scaled Lloyd-Max), a family of training-free vector quantization codecs compressing embeddings to 1--4 bits per dimension,...

By Rastislav Lenhardt, Teodora Dobos, Thomas Vecchiato, Jiri Isa, Igor Ginzburg

rag efficiency benchmarks safety

arXiv Machine Learning

Sep 1

In-Cell Learning: Language Models That Update Their Own Weights in Sequence Without Changing the File They Ship

arXiv:2608.20873v3 Announce Type: replace Abstract: A 4-bit quantized weight specifies a rounding cell rather than a single full-precision value. We introduce in-cell learning, a paradigm for writing...

By Zifeng Liu, Yaxin Lu, Xuanhan Wu, Zhiyong Du, Yiming Mao, Zhenhe Wang, Wenqi Shi, Zhengkun Jing, Linwei Liu

llms efficiency

arXiv Machine Learning

Sep 1

Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimization

arXiv:2608.30141v1 Announce Type: cross Abstract: Preference optimization is widely used to align large language models with human preferences, but preference-data composition may also influence priv...

By Dishu Yang, Jingjing Liu, Jize Li

llms reinforcement-learning efficiency safety

arXiv Machine Learning

Sep 1

When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction

arXiv:2608.30086v1 Announce Type: cross Abstract: Credit-default prediction is an important task in financial decision making. Traditional methods use fitted classifiers such as logistic regression a...

By Rishi Datta, Lavanya Prahallad

llms efficiency

arXiv Machine Learning

Sep 1

ADMM-Q: An Improved Hessian-based Weight Quantizer for Post-Training Quantization of Large Language Models

arXiv:2605.11222v2 Announce Type: replace Abstract: Quantization is an effective strategy to reduce the storage and computation footprint of large language models (LLMs). Post-training quantization (...

By Ryan Lucas, Mehdi Makni, Xiang Meng, Adam Deng, Rahul Mazumder

llms efficiency

arXiv Machine Learning

Sep 1

SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

arXiv:2608.28911v1 Announce Type: new Abstract: The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length....

By Daeha Lee, Do-Hyung Kim, Jae-Hong Kim

llms efficiency

arXiv Machine Learning

Sep 1

Sparse Koopman Autoencoders Identify Local Dynamical Regimes in Multibasin Systems

arXiv:2608.29057v1 Announce Type: new Abstract: Koopman autoencoders (KAEs) seek a higher-dimensional latent representation in which nonlinear dynamics evolve linearly. However, many interesting syst...

By Aidan Li, Uday Kiran Reddy Tadipatri, Mahan Fathi, Sarath Chandar, Ross Goroshin

rag efficiency

arXiv Machine Learning

Sep 1

Generalist Graph Anomaly Detection via Prototype-Based Distillation

arXiv:2605.26857v2 Announce Type: replace Abstract: Driven by the pressing demand for graph anomaly detection (GAD) in high-stakes domains, the generalist GAD paradigm, which trains a single detector...

By Yiming Xu, Zihan Chen, Zhen Peng, Song Wang, Bin Shi, Bo Dong, Chao Shen

efficiency safety

arXiv Computer Vision

Sep 1

LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck

arXiv:2606.13061v3 Announce Type: replace Abstract: Reasoning-driven universal multimodal embedding has advanced rapidly by introducing Chain-of-Thought (CoT) reasoning into the embedding pipeline. D...

By Peixi Wu, Biao Yang, Feipeng Ma, Bosong Chai, Bo Lin, Wei Yuan, Fan Yang, Tingting Gao, Hebei Li, Xiaoyan Sun

rag diffusion efficiency multimodal

arXiv Computer Vision

Sep 1

FLM: Frequency-Aware Language Models for Generative Image Compression

arXiv:2608.28687v1 Announce Type: new Abstract: Generative models have significantly improved the performance ceiling of image lossy compression at low bitrates by exploiting learned priors. However,...

By Jiarun Chen, Kejun Wu, Li Li, Chengtao Cai, Zhengguo Li, Chia-Wen Lin

llms nlp fine-tuning efficiency

arXiv Machine Learning

Sep 1

PairAlign: A Framework for Autoregressive Tokenization via Self-Alignment with Applications to Audio Tokenization

arXiv:2605.06582v4 Announce Type: replace Abstract: Modern learning systems represent perceptual signals with continuous vectors, but comparison, retrieval, memory, alignment, and reasoning are often...

By Adhiraj Banerjee, Vipul Arora

nlp efficiency safety

arXiv Machine Learning

Sep 1

Spectral Analysis for Sparse Matrix Computation: Insights and Potential

arXiv:2608.29362v1 Announce Type: cross Abstract: Sparse computations are fundamental to scientific computing, graph analytics, and machine learning, yet their performance is highly sensitive to the...

By Ruifeng Zhang, Xipeng Shen

llms efficiency benchmarks

arXiv Computation and Language

Sep 1

Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

arXiv:2606.10537v2 Announce Type: replace Abstract: Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with...

By Jing Xiong, Qi Han, Shansan Gong, Yunta Hsieh, Boyuan Zheng, Chengyue Wu, Chaofan Tao, Chenyang Zhao, Ngai Wong

llms diffusion efficiency benchmarks

arXiv Machine Learning

Sep 1

Delta-AI: Local objectives for amortized inference in sparse graphical models

arXiv:2310.02423v3 Announce Type: replace Abstract: We present a new algorithm for amortized inference in sparse probabilistic graphical models (PGMs), which we call $\Delta$-amortized inference ($\D...

By Jean-Pierre Falet, Hae Beom Lee, Esmeralda S. Whitammer, Chen Sun, Dragos Secrieru, Thomas Jiralerspong, Dinghuai Zhang, Guillaume Lajoie, Yoshua Bengio

agents efficiency

arXiv AI

Sep 1

TopGQ: Fast GNN Post-Training Quantization Leveraging Topology Information

TopGQ is a post‑training graph neural network (GNN) quantization framework that reduces quantization overhead by using dual‑axis scale absorption, which merges one dimension into the adjacency matrix for activation quantization. It also introduces TopPIN, a proxy for nodes’ local structure, to group nodes with similar topology during quantization. Experiments demonstrate that TopGQ cuts quantization time by an order of magnitude while maintaining accuracy.

By Dain Kwon, Kanghyun Choi, Hyeyoon Lee, Sunjong Park, Seoyong Lee, Sukjin Kim, Jinho Lee

efficiency

arXiv Machine Learning

Sep 1

Sharp Restricted Isometry Thresholds for Global Minima of Rank-Restricted Matrix LASSO

arXiv:2608.29018v1 Announce Type: cross Abstract: We determine the sharp restricted isometry threshold for recovery at global minima of the rank-restricted matrix LASSO. For target rank $r_{\star}$,...

By Richard Y. Zhang

efficiency

arXiv Machine Learning

Sep 1

Learning the Geometry of Admissible Hypotheses through Inductive Bias in Training Distributions

arXiv:2608.31028v1 Announce Type: cross Abstract: Scientific discovery often requires reasoning over competing hypotheses that are consistent with experimental observations. For mixed-variable and co...

By James Crowley, Faez Ahmed, Anton van Beek

rag efficiency benchmarks safety

arXiv Machine Learning

Sep 1

Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware

arXiv:2608.30439v1 Announce Type: cross Abstract: Inference with transformer-based large language models (LLMs) is often limited by the memory-bound KV cache and quadratic attention cost. State-space...

By Simon Richter, Ruhai Lin, Jason Yik, Taylor Kergan, Rui-Jie Zhu, Farshad Moradi, Jason Eshraghian

llms efficiency

Get the daily briefing