The paper introduces FACTS, a structured Fisher Approximation for compressing Vision Transformers (ViTs) using Fisher-weighted SVD, which enforces token‑local aggregation while preserving within‑token activation‑gradient dependence. It also presents Constrained Rank Search (CoRS) to optimize layer‑wise rank allocation under a fixed FLOP budget. Experiments on ViTs and hybrid architectures show that FACTS improves accuracy‑efficiency trade‑offs, outperforming the strongest SVD baseline by up to +5.8 percentage points on Swin‑B without requiring finetuning.
Vision-Language Models (VLMs) and Vision-Language-Action (VLA) models process hundreds of visual tokens, making visual representations a major computational bottleneck. We argue that visual token prun...
arXiv:2606. 00573v1 Announce Type: new Abstract: Vision-language models (VLMs) deliver strong multimodal reasoning capabilities, but their large computational cost and high parameter counts make deployment challenging on resource-constrained devices.
The paper introduces LASER, a loss‑aware low‑rank compression framework for vision‑language models. It derives a curvature‑weighted SVD objective from a second‑order loss approximation and uses Kronecker‑factored Fisher information to guide decomposition toward downstream performance. LASER also proposes a cross‑layer rank allocation strategy based on calibration gradients and extends compression to feed‑forward networks with a hybrid SVD‑quantization scheme, achieving over 2.3× decoding speedup while maintaining accuracy under low‑precision inference.
arXiv:2606.21847v2 Announce Type: replace-cross Abstract: Low-rank decomposition is a promising compression paradigm for large language models (LLMs), yet its effectiveness hinges on rank budget allo...
The paper introduces Layer-wise Error Modeling Search (LEMS) and KFAC-SVD to improve low-rank SVD-based compression of large language models. LEMS addresses global rank allocation by using a layer-wise error surrogate and an ILP formulation, while KFAC-SVD enhances decomposition quality by employing token-wise statistics to avoid rank collapse. Experiments on Mistral, Qwen3, and Llama 3 show LEMS yields up to 4.8 pp zero‑shot accuracy gains and KFAC‑SVD achieves an average perplexity improvement of 15%.
arXiv:2505. 17974v2 Announce Type: replace-cross Abstract: The Fisher information is a fundamental concept for characterizing the sensitivity of parameters in neural networks.
arXiv:2510. 05544v2 Announce Type: replace-cross Abstract: Large language models (LLM) and vision-language models (VLM) have achieved state-of-the-art performance, but they impose significant memory and computing challenges in deployment.
arXiv:2509. 25136v3 Announce Type: replace Abstract: Activation-aware low-rank factorization techniques yield strong compression results but are generally confined to linear layers, while existing whitening-based theory typically makes an implicit full-rank assumption on activations.
arXiv:2602.02848v2 Announce Type: replace Abstract: Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD...
arXiv:2604. 11530v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) have revolutionized multi-modal learning by jointly processing visual and textual information.
arXiv:2606. 07098v1 Announce Type: cross Abstract: We present SigmaScale, a method for learning auxiliary scaling matrices $S$ to aid truncated Singular Value Decomposition (SVD) based Large Language Model (LLM) compression.
arXiv:2610.17412v1 Announce Type: cross Abstract: Vision Transformers incur high computational costs when processing long visual token sequences. Existing reduction methods typically rank tokens inde...