arXiv AI November 6, 2026 By Moonseok Choi, Jongwon Jeong, Minki Kang, Giung Nam, Juho Lee

SKIM: Pruning Large Language Model Agents via Selective Knowledge Informed Masking

SKIM (Selective Knowledge-Informed Masking) is a pruning and distillation framework designed for large language model agents that operate under structured generation, where grammar constraints limit output to valid tokens. The method partitions token positions into context, decision, and format categories, using category-aware pruning to preserve parameters most relevant to decision positions and an on-policy distillation objective that aligns student and teacher on valid tokens with category-aware feature matching. Evaluations on tool-calling benchmarks and multi-step agent loops across various model scales and sparsity levels show that SKIM consistently outperforms baseline compression methods, especially in high-sparsity regimes where conventional techniques falter.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

Aug 31

Pruning Laws for Large Language Models

arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...

By Ayan Sengupta, Siddhant Chaudhary, Tanmoy Chakraborty

llms efficiency

arXiv Computation and Language

Oct 23

Beyond Prompt and Parser Engineering: Soft-Prompt Tuning for Automated LLM Evaluation

The paper introduces soft‑prompt tuning as a lightweight, automated method for evaluating large language models (LLMs). By optimizing only ten soft‑prompt vectors—about 0.0006% of a 7B model’s parameters—within 150 steps, the approach adapts models to specific benchmark formats, improving format compliance while preserving knowledge accuracy. Experiments across seven models and eleven datasets show that this method matches strong evaluation‑harness baselines, enhances evaluation signals on challenging datasets, and yields cleaner outputs that align better with exact‑match scoring and independent LLM judges.

By Selen Erkan, Bastian Boll, Kristian Kersting, Bj\"orn Deiseroth, Letitia Parcalabescu

llms benchmarks

arXiv Machine Learning

Jul 30

Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation

arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.

By Xin Yu, Cong Xie, Xunmei Liu, Tiantian Fan, Lingzhou Xue, Zhi Zhang

llms fine-tuning efficiency

arXiv AI

Oct 15

LLM Agents for Automated Post-Training via Structured Recipe Optimization

arXiv:2610.18572v1 Announce Type: new Abstract: Automated post-training transforms pre-trained large language models (LLMs) into models capable of handling diverse domains by automatically constructi...

By Ruohong Liu, Bohan Tang, Lekang Jiang, Jack Umenberger, Yiwen Guo

llms agents benchmarks

arXiv Machine Learning

Aug 14

Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization

arXiv:2608. 12953v1 Announce Type: cross Abstract: Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and often fail to precisely meet target compression budgets.

By Palaash Goel, Ayan Sengupta, Akshay Nambi, Tanmoy Chakraborty

llms efficiency benchmarks

Hugging Face Trending Papers

Oct 22

Harnessing Language Models as Optimization Modeling Agents

The paper introduces two lightweight inference harnesses, OPTICYCLE and OPTISTATE, designed to enable large language models to translate natural-language optimization problems into solver programs without additional training. Evaluated across nine benchmarks and 11 Qwen variants (0.6B–27B parameters), the harnesses consistently improve macro accuracy, with execution-guided revision boosting performance by up to 49.46 percentage points. Despite their simplicity, both harnesses outperform six adapted agentic workflows and four post-trained optimization models of comparable size, though the benefit of explicit formulation state varies by model.