arXiv AI

ACE-Compression: An Ability-Centric Evaluation of Compressed Large Language Models

ACE-Compression introduces an ability‑centric evaluation framework for compressed large language models, moving beyond traditional task‑performance metrics. It probes four deployment‑relevant dimensions—procedural reasoning, knowledge grounding, social cognition, and persona fidelity—to determine which core abilities survive compression. The study finds that compression redistributes ability profiles, with coarse signals persisting while fine‑grained abilities become fragile, and that quantization generally retains more abilities than pruning.

arXiv Machine Learning

Oct 5

Capability Scaling-Down Laws for LLM Compression

The paper presents a systematic study of how different compression techniques—pruning, quantization, and distillation—affect the capabilities of large language models (LLMs) in tasks such as mathematics, code generation, and question answering. It introduces a framework that measures capability loss and relates it to factors like model size, training stage, and compression settings, yielding simple predictive relations that generalize across unseen configurations. The authors demonstrate that sharing density responses across pruning levels can dramatically reduce the number of measurements needed, and that their predictive models closely match regression results while offering efficient decision guidance for compression method selection.

By Xueqi Cheng, Liang Wu, Kelly Wan, Liangjie Hong, Yushun Dong

llms nlp efficiency

arXiv Computation and Language

Sep 15

Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories

arXiv:2609.13154v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have made prompts increasingly large and complex. Techniques such as chain-of-thought reasoning (Wei et...

By Shamin Chokshi

llms nlp efficiency benchmarks

arXiv AI

Sep 24

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

arXiv:2609.27510v1 Announce Type: cross Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and...

By Kaifeng Tan, Yudong Li, Linlin Shen

llms benchmarks

arXiv AI

Sep 1

Capability-Stratified Degradation in Ternary Language Models

arXiv:2608.28809v1 Announce Type: new Abstract: Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, a...

By Anirudh Malik, M Sparsh Mehra, Poojith Devan

llms fine-tuning benchmarks

arXiv Machine Learning

Oct 8

Latent Performance Profiling of Large Language Models

arXiv:2605.30018v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of the...

By Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya, Partha Pratim Chakrabarti, Amlan Chakrabarti, Supratik Chakraborty, Partha Pratim Das, Lipika Dey, Richa Singh, Mayank Vatsa

llms benchmarks safety

Hugging Face Trending Papers

Sep 23

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results....

llms benchmarks

arXiv Machine Learning

Jun 29

FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

arXiv:2606. 27866v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models.

By Fan Mo, Yuxuan Han, Geng Zhang, Wangbo Zhao, Yang You

llms fine-tuning efficiency

arXiv AI

Jun 24

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

arXiv:2606. 24083v1 Announce Type: cross Abstract: "Talk short.

By Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt

llms benchmarks

arXiv AI

Oct 15

SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications

arXiv:2610.14918v1 Announce Type: new Abstract: Evaluating an agent's ability to strictly satisfy diverse and procedurally complex requirements is essential to determining its practical reliability....

By Zhengxi Li, Junyi Zhou, Qiyuan Zhang, Chuhan Wu, Fuyuan Lyu, Le Tian, Xiao Zhou, Chen Ma, Yufei Wang

agents benchmarks

arXiv AI

Jun 15

Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2

arXiv:2512. 22671v3 Announce Type: replace-cross Abstract: Structured width pruning of GLU-MLP layers in Llama-3.

By Pere Martra

llms efficiency benchmarks

arXiv AI

Jun 17

In-Context Environments Induce Evaluation-Awareness in Language Models

arXiv:2603. 03824v2 Announce Type: replace Abstract: Humans often become more self-aware under threat, yet can lose self-awareness when absorbed in a task; we hypothesize that language models exhibit environment-dependent \textit{evaluation awareness}.

By Maheep Chaudhary

llms benchmarks safety

arXiv AI

Aug 17

Depth-Aware Sensitivity Analysis of Mixture-of-Experts Models via Magnitude-Based Expert Masking

arXiv:2608. 13565v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) architectures scale large language models (LLMs) while preserving computational efficiency through sparse activation.

By Pradeep Kumar Sharma, Shantanu Godbole, Hritvik Shrivastava

llms efficiency benchmarks

Related stories