Inference efficiency

Quantization, distillation, pruning and serving work aimed at the same accuracy for less memory, latency and money.

3,248 stories · RSS feed

arXiv Machine Learning

Jul 13

LDPKiT: Superimposing Remote Queries for Privacy-Preserving Distillation

arXiv:2405. 16361v4 Announce Type: replace Abstract: To protect privacy in regulated domains such as healthcare and finance, model owners may allow only remote API access while keeping both the training data and model parameters private.

By Kexin Li, Aastha Mehta, David Lie

diffusion efficiency safety

arXiv AI

Jul 13

STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU

arXiv:2607. 09385v1 Announce Type: cross Abstract: The growing adoption of large language model-based agents within operating system workflows has increased the importance of energy-efficient inference on laptop-class systems-on-chip (SoCs).

By Victor J. B. Jung, Gagandeep Singh, Joseph Melber, Kristof Denolf, Francesco Conti, Luca Benini

llms agents efficiency safety

arXiv AI

Jul 13

Signed Symmetric Quantization for Few-Bit Integers

arXiv:2607. 08779v1 Announce Type: cross Abstract: The signed integer alphabet contains one more negative representable value than positive.

By Ian Colbert, Eashan Dash, Pablo Monteagudo-Lago, Juan Amboage, Srinidhi N, Giuseppe Franco, Nicholas J. Fraser, Arun Ramachandran

llms efficiency

arXiv AI

Jul 13

A Sovereign, Open-Source Foundation Model for German and English

arXiv:2607. 09424v1 Announce Type: cross Abstract: We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English.

By The Soofi-Team, :, Benedikt Droste, David Fitzek, Ruben H\"arle, Lukas Helff, Maximilian Idahl, Alex Jude, Abbas Goher Khan, Maurice Kraus, Timm Ruland, Richard Rutmann, Sebastian Sztwiertnia, Markus Frey, Daniil Gurgurov, Jan Pfister, Tom R\"ohr, Sebastian von Rohrscheidt, J\"org Bienert, Nicolas Flores-Herr, Simon Gottschalk, Andreas Hotho, Kristian Kersting, Joachim K\"ohler, Alexander L\"oser, Wolfgang Nejdl, Simon Ostermann, Jan Plogsties, Patrick Putzky, Mehdi Ali, Michael Fromm, Max L\"ubbering

llms efficiency benchmarks

arXiv AI

Jul 13

Multi-Attribute Steering of Language Models via Targeted Intervention

arXiv:2502. 12446v3 Announce Type: replace-cross Abstract: Inference-time intervention (ITI) has emerged as a promising method for steering large language model (LLM) behavior in a particular direction (e.

By Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal

llms nlp fine-tuning efficiency safety

arXiv Machine Learning

Jul 13

Self-Improving Neural-Guided Pruning: A Graph Neural Network Framework for Scalable Mixed Bundle Pricing

arXiv:2509. 22557v5 Announce Type: replace Abstract: Mixed bundle pricing is a classic revenue management problem arising in industries such as e-commerce, tourism, and video games.

By Liangyu Ding, Chenghan Wu, Guokai Li, Zizhuo Wang

efficiency

arXiv AI

Jul 13

Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices

arXiv:2607. 08786v1 Announce Type: cross Abstract: With the growing deployment of large language models (LLMs), LLM inference cost has become a key challenge.

By Tao Lu, Haoyu Wang, Zonghui Wang, Keshen Xiang, Jiaheng Zhang, Wenzhi Chen

llms efficiency

arXiv AI

Jul 13

A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions

arXiv:2607. 08776v1 Announce Type: cross Abstract: Despite the success of knowledge distillation (KD) in Large Language Models (LLMs), the underlying mechanism behind its efficacy remains unclear.

By Qingzhuo Wang, Ruiyang Qin, Zhenxin Qin, Wen Shen, Zhihua Wei

llms efficiency benchmarks

arXiv AI

Jul 13

KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling

arXiv:2607. 09153v1 Announce Type: new Abstract: Process Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLM-based multi-agent systems.

By Peng Kuang, Haibo Jin, Xiaoyu Han, Yanli Wang, Xiaopeng Yuan, Ye Yu, Kaidi Xu, Haohan Wang

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Jul 13

Single-Frame Point-Pixel Registration via Supervised Cross-Modal Feature Matching

arXiv:2506. 22784v2 Announce Type: replace-cross Abstract: Point-pixel registration between LiDAR point clouds and camera images is a fundamental yet challenging task in autonomous driving and robotic perception.

By Yu Han, Zhiwei Huang, Yanting Zhang, Fangjun Ding, Shen Cai, Xiaoyu Tang, Yanchao Dong, Rui Fan

agents robotics efficiency multimodal benchmarks

arXiv AI

Jul 13

ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts

arXiv:2607. 09526v1 Announce Type: cross Abstract: Foundation models are reshaping computational pathology, yet their capabilities remain shaped by pretraining objectives, data sources, and spatial scales, fragmenting complementary expertise across separate backbones.

By Jiawen Li, Tian Guan, Huijuan Shi, Xitong Ling, Mingxi Fu, Anjia Han, Chao He, Yonghong He

efficiency multimodal

arXiv AI

Jul 13

Integrating Large Language Models and Graph Convolutional Networks for Semi-Supervised Image Classification

arXiv:2607. 09104v1 Announce Type: cross Abstract: While the growing availability of image data has driven significant advances, labeling datasets remains costly and time-consuming.

By Camila Piscioneri Magalh\~aes, Lucas Pascotti Valem

llms computer-vision efficiency multimodal

arXiv AI

Jul 13

REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression

arXiv:2508. 15806v2 Announce Type: replace-cross Abstract: The growing sequence length of large language models poses significant challenges for key-value (KV) caches.

By Mengjie Li, Yuan Feng, Xike Xie, William J. Song

llms efficiency benchmarks safety

arXiv AI

Jul 13

IB-Flow: Information Bottleneck-Guided CFG Distillation for Few-Step Text-to-Image Generation

arXiv:2607. 09133v1 Announce Type: cross Abstract: While large-scale text-to-image generative models have achieved unprecedented visual performance, their inherent reliance on multi-step iterative solvers incurs severe inference latency.

By Yiting Wang, Jingyi Zhang, Wenhu Zhang, Ke Chao, Yves Liang, Kun Cheng, Kang Zhao

diffusion efficiency benchmarks

arXiv AI

Jul 13

PAC-ACT: Post-training Actor-Critic for Action Chunking Transformers

arXiv:2607. 09590v1 Announce Type: cross Abstract: Precision industrial contact manipulation requires reliable robot policies under pose perturbations and contact-force constraints.

By Yujie Pang, Zudong Li

llms rag robotics fine-tuning efficiency multimodal benchmarks safety

arXiv Machine Learning

Jul 13

EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems

arXiv:2607. 09063v1 Announce Type: new Abstract: Edge devices are increasingly utilized for deploying deep learning applications on embedded systems.

By Shuo Huai, Hao Kong, Shiqing Li, Xiangzhong Luo, Ravi Subramaniam, Christian Makaya, Qian Lin, Weichen Liu

efficiency benchmarks

arXiv Machine Learning

Jul 13

COBS: Cumulant Order Block Sparse Attention

arXiv:2607. 09052v1 Announce Type: new Abstract: Block sparse attention is a hardware friendly way to alleviate the key-value (KV) cache read bottleneck in large language models (LLMs).

By Alexander Tian, Aditya Ghai, Sanjit Neelam, Zaal Vasania, Akshay Mishra

llms efficiency benchmarks

arXiv AI

Jul 13

Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference

arXiv:2607. 09520v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are the perceptual backbone of embodied AI, but their energy footprint on edge hardware remains poorly understood.

By Junfei Zhan, Haoxun Shen, Mingang Guo, Zixuan Huang, Tengjiao He

llms robotics efficiency multimodal

arXiv Machine Learning

Jul 13

Federated Low-Rank Koopman Learning for Multivariate Time-Series Anomaly Detection in IoT Systems

arXiv:2607. 08978v1 Announce Type: new Abstract: Distributed IoT systems generate multivariate time-series streams for monitoring physical assets, servers, and embedded sensing platforms.

By Tung-Anh Nguyen, Van-Phuc Bui, Anh Tuyen Le, Kim Hue Ta, Minh Thuy Le, J. Andrew Zhang, Xiaojing Huang

efficiency benchmarks

arXiv AI

Jul 13

Blockchain-Linked Auditable Decision Management for Telecom/IoT Fraud-Control Requests

arXiv:2607. 09259v1 Announce Type: cross Abstract: Telecom fraud-control studies often stop at detector-level classification, but deployment use requires request-level policy resolution, lifecycle traceability, and auditability.

By Saviz Changizi, Nasibeh Mohammadzadeh, Mohammad Shojafar, Rahim Tafazolli

llms fine-tuning efficiency

Get the daily briefing