arXiv AI
Scaling Influence Functions in LLMs through Eigenbasis-Corrected One-Bit Gradient Projection
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 8
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
arXiv:2606. 07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression.
arXiv Machine Learning
Aug 31
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
The paper investigates the often-overlooked scale vectors in large language models, showing that despite their tiny size they are crucial for pre‑training performance. The authors provide theoretical insights that scale vectors mainly aid optimization rather than expressivity, and they analyze how weight decay affects different normalization layers. Building on these findings, they propose lightweight improvements—branch‑specific heterogeneity, better placement, and magnitude‑direction reparameterization—that consistently reduce loss across a range of model sizes and training settings.
arXiv Machine Learning
Jul 17
Stabilizing Native Low-Rank LLM Pretraining
arXiv:2602. 12429v2 Announce Type: replace Abstract: Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges.
arXiv AI
Aug 26
Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression
The paper introduces the "Compression Trinity," a unified framework that jointly applies sparsity, quantization, and low‑rank approximations to compress large language models. It presents several methods—MKOR, SLoPe, OPTIMA, PATCH, and SLiM—that leverage these three pillars to accelerate training, reduce memory bandwidth, and recover accuracy, achieving significant speedups and accuracy gains over existing techniques. The results demonstrate that combining all three compression strategies is essential for efficient, scalable, high‑performance LLM deployment.
arXiv Machine Learning
Sep 30
Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression
arXiv:2602.02848v2 Announce Type: replace Abstract: Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD...
arXiv Machine Learning
Jun 2
ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression
arXiv:2606. 00494v1 Announce Type: new Abstract: Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA) constitute the standard pipeline for efficient Large Language Model (LLM) deployment.