The paper "CompressionBench: How can your coding agents compress LLMs?" investigates whether autonomous coding agents can compress large language models (LLMs) while staying within a specified storage budget. The study frames compression as a terminal task where agents receive a source checkpoint, storage limit, inference environment, and time budget, then generate compressed weights and executable inference code, guided by perplexity feedback and verified by independent checks. Across six agents and six model-size settings, 29 of 36 submissions passed verification, achieving an average 3.4% increase in perplexity over BF16 and 77.33% downstream accuracy, with detailed experiments revealing strengths and failure points in agent-driven compression.
By Xinrui Zhong, Zhifeng Li, Zhiming Wang, Yongqi Liu, Shanglin Yang, Yang Zhou, Zikun Li, Zhuoming Chen, Beidi Chen
arXiv:2608. 05499v1 Announce Type: cross Abstract: Modern deep neural networks achieve strong performance, but their scale makes them costly and slow, especially on resource-constrained edge devices.
By Sadegh Jafari, Mohiuddin Bilwal, Fan Zhou, Brian Gelder, Ali Jannesari
FairLMs is a Python library designed to streamline fairness research in language models by unifying bias measurement, mitigation, and evaluation evidence. It offers 33 intrinsic and extrinsic metrics, 14 mitigation components across four intervention categories, 14 diagnostic tools, adapters for major Transformer architectures and hosted APIs, and benchmark loaders. The library enforces explicit declarations of model capabilities and input requirements, ensuring compatibility and reproducibility across components and datasets.
By Jiale Zhang, Michael Larionov, Zichong Wang, Zhipeng Yin, Wenbin Zhang
Modern deep neural networks achieve strong performance, but their scale makes them costly and slow, especially on resource-constrained edge devices. Pruning and quantization address this, but rely on manual, expert choices and on algorithms that are hard to apply across architectures.
GAMMA is a post‑training framework that learns module‑wise precision preferences for mixed‑precision quantization of large language models. It optimizes a teacher‑forced hidden‑state reconstruction objective under an augmented Lagrangian constraint and then projects the learned preferences into exact budget‑feasible discrete assignments via integer programming. Because the learned preferences encode a stable sensitivity ranking, a single training run can be reused for any deployment budget, reducing per‑budget adaptation from hours to minutes and outperforming fixed‑precision baselines and search‑based methods on Llama and Qwen models.
By Zhangyang Yao, Haiyan Zhao, Haoyu Wang, Xu Han
AutoCompress is a hardware‑guided model compression framework that integrates latency, throughput, and memory constraints directly into differentiable architecture search. It jointly optimizes global and per‑layer structures to produce heterogeneous architectures, using serving‑engine cost models validated against real end‑to‑end measurements. In evaluations on long‑context coding, datacenter serving, and in‑game interaction, AutoCompress outperforms the Minitron pruning framework, improving average accuracy by up to 6.57 percentage points on RTX 5070 and compressing Nemotron 3.5 Super from 120B to 74B parameters while retaining 95% of its benchmark score.
By Sheldon Liang, Ali Taghibakhshi, Saurav Muralidharan, Sharath Turuvekere Sreenivas, Ashwath Aithal, Pavlo Molchanov, Nima Tajbakhsh
arXiv:2605.29350v2 Announce Type: replace
Abstract: Sparse Mixture-of-Experts (MoE) models scale model capacity through sparse activation, but storing their full parameters remains costly in GPU memo...
By Yilun Yao, Jiaming Pan, Elsie Dai, Peizhuang Cong, Chao-Hsuan Liu, Yaoming Li, Tong Yang
arXiv:2605. 17160v2 Announce Type: replace-cross Abstract: Model quantization is widely used to reduce memory, latency, and deployment cost, and is typically judged by whether predictive accuracy is preserved.
By Chaymae Yahyati, Ismail Lamaakal, Khalid El Makkaoui, Ibrahim Ouahbi
ACSeL is a framework for selecting hardware‑constrained configurations of large language models that satisfy memory and latency budgets while maximizing generation quality. It uses a lightweight quality surrogate based on Relative Model Quality Degradation (RMQD), a few‑shot median calibration (FMC) to refine latency estimates, and a BandPareto search algorithm to explore mixed‑precision layerwise settings. Experiments on NVIDIA RTX A6000 and Jetson Thor devices show that FMC improves deployment feasibility and that BandPareto + FMC achieves the best feasibility and lowest RMQD gaps to optimal configurations.
By Bokun Wang, Hung-Yueh Chiang, Diana Marculescu
The paper presents a Pareto atlas of LLM inference optimizations, mapping cost, quality, and latency trade‑offs for Qwen2.5‑7B‑Instruct on L4, A100, and H100 GPUs. Using 54 measured configurations and a calibrated simulator, it identifies 18 of 36 setups on the Pareto frontier, showing that combined methods outperform single ones. Quality tests reveal that AWQ 4bit and FP8 weights offer significant latency reductions while largely preserving accuracy, but naive FP8 KV caching fails to answer any questions correctly.
By Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
The paper introduces BestRand, a benchmark‑compression technique that retains the efficiency of random sampling while dramatically cutting the high‑error tail of estimation errors. By analytically approximating the error distribution when estimating full‑benchmark scores from subsets, the authors show that random sampling is surprisingly effective but unstable due to rare large errors. BestRand reduces high‑error risk by 50.2 % and overall estimation error by 16.4 % on average, while cutting construction cost by almost half, and it also lowers out‑of‑distribution estimation error by 21.8 % across various benchmarks.
By George Ling, Shanshan Zhong, Lijin Yang, Hao Yang, Nanxiang Li, Zhongzhan Huang
arXiv:2609.07966v1 Announce Type: new
Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...
By Michael Wang, Keith Li, Roozbeh Bostandoost