arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.
The paper introduces αp-LoRA, a rank-allocation strategy for low-rank adaptation (LoRA) in large language models that uses π-regularization (0 < p < 1) to induce sparsity in rank-one components. By regularizing the energy of each component, redundant parts are encouraged to vanish while important ones are retained, and the authors derive a proximal subproblem that reduces the matrix optimization to a two‑dimensional thresholding criterion. Experiments on natural language understanding and question‑answering tasks show that αp-LoRA achieves performance competitive with existing LoRA baselines.
arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
The paper introduces ρ_p-LoRA, a rank-allocation strategy for low-rank adaptation (LoRA) in large language models that uses ρ_p regularization (0 < p < 1) to induce sparsity in rank-one components. By regularizing the energy of each component, redundant parts are encouraged to vanish while important ones are retained, leading to an implicit thresholding criterion derived from a two-dimensional proximal subproblem. Experiments on natural language understanding and question-answering tasks show that ρ_p-LoRA achieves performance comparable to existing LoRA baselines.
arXiv:2603. 13418v2 Announce Type: replace Abstract: Structured pruning is widely applied to compress large language models (LLMs), but its performance depends heavily on how neuron importance is estimated.
arXiv:2505. 18877v4 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix.
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization.
arXiv:2509. 14230v2 Announce Type: replace Abstract: While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities.
arXiv:2602. 01997v3 Announce Type: replace-cross Abstract: Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning.
The paper introduces REP‑LIE, a resource‑efficient pruning method for Transformer models that estimates weight importance using gradients from LoRA low‑rank matrices, avoiding full gradient computation. It incorporates a stability score to iteratively prune unimportant parameters and then fine‑tunes the pruned model with lightweight updates, eliminating the need for full‑parameter optimization. Experiments on medium‑scale encoders and large‑scale generative models such as LLaMA‑7B and Mistral‑7B show that REP‑LIE achieves competitive performance compared to existing pruning approaches.
arXiv:2607. 20205v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models.
arXiv:2507. 01900v3 Announce Type: replace-cross Abstract: Pruning is a highly effective approach for compressing large language models (LLMs), significantly reducing inference latency.