The paper introduces EDGC, an entropy-driven dynamic gradient compression framework designed to reduce communication overhead during large language model training. EDGC adapts compression ranks based on gradient entropy, using efficient entropy estimation, a theoretical entropy-to-rank model, and window-based rank adjustments across pipeline stages. Experiments on GPT‑2 models with 2.5B and 12.1B parameters on 32‑V100 and 64‑H100 GPU clusters demonstrate up to 46.45% lower communication latency and a 16.13% reduction in end‑to‑end training time while preserving model quality.
arXiv:2609.25537v1 Announce Type: new Abstract: Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing laten...
arXiv:2509. 14230v2 Announce Type: replace Abstract: While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities.
The paper reports six submissions by the ESTS team to the WMT26 Model Compression Shared Task for English–Simplified Chinese and English–Egyptian Arabic. Each submission offers three compression operating points derived from GPT‑OSS‑20B, using routing‑informed expert pruning, cross‑lingual routing divergence for capacity allocation, and MXFP4 quantization of retained expert projection weights. The resulting models, ranging from 4.186 B to 7.770 B parameters, are fine‑tuned on GPT‑5.1 synthetic data and evaluated internally with xCOMET‑XL.
The paper introduces Ladder Side Tuning (LST), a parameter‑efficient fine‑tuning method that adds a lightweight side network to large language models. LST matches QLoRA’s compute scaling while halving peak memory usage, enabling 7B‑parameter models to be fine‑tuned on a single 12 GB GPU with 2k‑token contexts without gradient checkpointing. The authors also present xLadder, a depth‑extended variant that increases effective depth through cross‑connections, allowing deeper reasoning without extra memory overhead.
The paper introduces BASP, a batch‑aware sequence parallelism method that partitions GPUs into disjoint groups based on micro‑batch size to reduce all‑to‑all communication. By localizing communication, BASP improves training efficiency for long‑context LLMs. Experiments on NVIDIA A100 clusters show up to 1.17‑1.31× faster end‑to‑end training on Llama and Qwen models while maintaining the same accuracy and memory usage.
The paper introduces the Tri‑Metric Router, a deterministic, training‑free policy that chooses among Raw, Neural, and Lexical pipelines for retrieval‑augmented generation on commodity GPUs. It uses three CPU‑side signals—spatial complexity, syntactic density, and type‑token ratio—to balance VRAM headroom and latency, calibrated on LongBench qasper. The method eliminates out‑of‑memory failures and improves alignment and F1 scores compared to always‑on lexical compression without extra VRAM or training costs.
arXiv:2602. 21788v2 Announce Type: replace-cross Abstract: Scaling long-context capabilities is crucial for Large Language Models (LLMs).
arXiv:2609.37842v1 Announce Type: cross Abstract: Influence functions estimate how individual training examples affect the behavior of large language models (LLMs). Analyzing how training data influe...
arXiv:2605. 28066v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable efficacy in text embedding, yet current adaptation methods like LoRA face significant bottlenecks in computational efficiency and cross-architecture transferability.
arXiv:2609.27510v1 Announce Type: cross Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and...
The paper proposes a layer-wise curriculum learning strategy for compressing large language models (LLMs). By partitioning the model into layer segments and starting training with easier tasks before progressing to harder ones, the method accelerates convergence and stabilizes knowledge transfer from teacher to student models. Additional techniques such as feature caching with multi-threading improve GPU utilization, leading to state‑of‑the‑art compression results and over 50% reductions in memory usage and training time on BERT and GPT‑2, while outperforming other pruning methods on LLaMA‑family and Qwen models.
The paper introduces the "Compression Trinity," a unified framework that jointly applies sparsity, quantization, and low‑rank approximations to compress large language models. It presents several methods—MKOR, SLoPe, OPTIMA, PATCH, and SLiM—that leverage these three pillars to accelerate training, reduce memory bandwidth, and recover accuracy, achieving significant speedups and accuracy gains over existing techniques. The results demonstrate that combining all three compression strategies is essential for efficient, scalable, high‑performance LLM deployment.