SKIM (Selective Knowledge-Informed Masking) is a pruning and distillation framework designed for large language model agents that operate under structured generation, where grammar constraints limit output to valid tokens. The method partitions token positions into context, decision, and format categories, using category-aware pruning to preserve parameters most relevant to decision positions and an on-policy distillation objective that aligns student and teacher on valid tokens with category-aware feature matching. Evaluations on tool-calling benchmarks and multi-step agent loops across various model scales and sparsity levels show that SKIM consistently outperforms baseline compression methods, especially in high-sparsity regimes where conventional techniques falter.
arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
The paper introduces soft‑prompt tuning as a lightweight, automated method for evaluating large language models (LLMs). By optimizing only ten soft‑prompt vectors—about 0.0006% of a 7B model’s parameters—within 150 steps, the approach adapts models to specific benchmark formats, improving format compliance while preserving knowledge accuracy. Experiments across seven models and eleven datasets show that this method matches strong evaluation‑harness baselines, enhances evaluation signals on challenging datasets, and yields cleaner outputs that align better with exact‑match scoring and independent LLM judges.
arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.
arXiv:2610.18572v1 Announce Type: new Abstract: Automated post-training transforms pre-trained large language models (LLMs) into models capable of handling diverse domains by automatically constructi...
arXiv:2608. 12953v1 Announce Type: cross Abstract: Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and often fail to precisely meet target compression budgets.
The paper introduces two lightweight inference harnesses, OPTICYCLE and OPTISTATE, designed to enable large language models to translate natural-language optimization problems into solver programs without additional training. Evaluated across nine benchmarks and 11 Qwen variants (0.6B–27B parameters), the harnesses consistently improve macro accuracy, with execution-guided revision boosting performance by up to 49.46 percentage points. Despite their simplicity, both harnesses outperform six adapted agentic workflows and four post-trained optimization models of comparable size, though the benefit of explicit formulation state varies by model.
arXiv:2609.13154v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have made prompts increasingly large and complex. Techniques such as chain-of-thought reasoning (Wei et...
arXiv:2505.19893v2 Announce Type: replace Abstract: Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce E...
Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and often fail to precisely meet target compression budgets. We present SNIPER, a two-stage structured pruning framework that solves a knapsack optimization over coarse-granularity components to yield conditionally optimal parameter allocations with respect to fixed importance estimates, followed by a fine-grained pruning stage to meet strict budget constraints.
OptSkills is an archetype‑centric agent that learns and reasons about optimization problems using large language models. It clusters problems by underlying archetypes, explores diverse modeling and solver configurations within each cluster, and distills successful trajectories into reusable workflow‑level skills. The system achieves state‑of‑the‑art accuracy on multiple datasets, outperforming prior methods on challenging benchmarks such as MIPLIB‑NL and OOD NLCO.
arXiv:2606. 12117v1 Announce Type: cross Abstract: Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.
arXiv:2510. 00615v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed as agents in dynamic real-world environments, where success depends on maintaining precise records of actions and observations.