arXiv AI
Jul 7
Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
arXiv:2607. 04371v1 Announce Type: new Abstract: We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment.
AutoCompress: Automated Hardware-Guided Model Compression
AutoCompress is a hardware‑guided model compression framework that integrates latency, throughput, and memory constraints directly into differentiable architecture search. It jointly optimizes global and per‑layer structures to produce heterogeneous architectures, using serving‑engine cost models validated against real end‑to‑end measurements. In evaluations on long‑context coding, datacenter serving, and in‑game interaction, AutoCompress outperforms the Minitron pruning framework, improving average accuracy by up to 6.57 percentage points on RTX 5070 and compressing Nemotron 3.5 Super from 120B to 74B parameters while retaining 95% of its benchmark score.
Related stories
arXiv AI
Sep 10
RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems
arXiv:2609.05760v1 Announce Type: cross Abstract: We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU enviro...
arXiv Machine Learning
Sep 17
Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs
The paper introduces the Tri‑Metric Router, a deterministic, training‑free policy that chooses among Raw, Neural, and Lexical pipelines for retrieval‑augmented generation on commodity GPUs. It uses three CPU‑side signals—spatial complexity, syntactic density, and type‑token ratio—to balance VRAM headroom and latency, calibrated on LongBench qasper. The method eliminates out‑of‑memory failures and improves alignment and F1 scores compared to always‑on lexical compression without extra VRAM or training costs.
arXiv AI
Jun 9
Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT
arXiv:2601. 20408v2 Announce Type: replace-cross Abstract: Enterprise LLM deployment faces a critical scalability challenge: organizations must optimize models systematically to scale AI initiatives within constrained compute budgets, yet the specialized expertise required for manual optimization remains a niche and scarce skillset.
arXiv AI
Oct 5
Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts
The paper introduces a hardware-software co‑design framework that compresses Mixture‑of‑Experts (MoE) model weights into low‑precision, hardware‑native sparse representations, enabling efficient execution on Sparse Tensor Cores (SpTCs). By relaxing discrete support selection through continuous reparameterization, the method jointly optimizes quantized weights and supports a router‑weighted reconstruction objective, achieving up to 4.35 percentage‑point gains in joint sparse‑quantization accuracy while retaining 96.09% of the original model’s performance. A custom grouped sparse GEMM kernel further boosts inference speed, outperforming NVIDIA’s baseline by up to 1.65× and reducing latency by up to 4.03× on B200 GPUs.
arXiv Machine Learning
6d ago
ACSeL: Adaptive Configuration Search for Hardware-Constrained LLM Inference
ACSeL is a framework for selecting hardware‑constrained configurations of large language models that satisfy memory and latency budgets while maximizing generation quality. It uses a lightweight quality surrogate based on Relative Model Quality Degradation (RMQD), a few‑shot median calibration (FMC) to refine latency estimates, and a BandPareto search algorithm to explore mixed‑precision layerwise settings. Experiments on NVIDIA RTX A6000 and Jetson Thor devices show that FMC improves deployment feasibility and that BandPareto + FMC achieves the best feasibility and lowest RMQD gaps to optimal configurations.
arXiv AI
Aug 21
From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG
arXiv:2608. 19535v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy.
arXiv Machine Learning
Aug 12
CARB: A Characterization-Guided Framework for CNN Inference Cost Prediction and Deployment Screening
arXiv:2608. 10506v1 Announce Type: cross Abstract: Accurate pre-deployment estimation of CNN inference cost--energy, latency, and peak memory--is increasingly critical as models are deployed on resource-constrained GPU platforms.
arXiv AI
Jun 9
End-to-End Context Compression at Scale
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
arXiv AI
Oct 31
EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models
EdgeCIM is a hardware-software co-design framework that accelerates autoregressive decoding of small language models on edge devices. It combines a CIM macro built in 65 nm with a tile-based mapping strategy to balance pipeline stages and reduce DRAM bandwidth bottlenecks. Benchmarks show that, at INT4 precision, EdgeCIM delivers up to 7.07× higher throughput than competing edge accelerators and averages 336.4 tokens/s with 173 tokens/J across a range of models up to 4 B parameters.
arXiv AI
Jul 24
Profiling Lightweight Large Language Models
arXiv:2607. 20806v1 Announce Type: new Abstract: Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments.
arXiv Machine Learning
Jun 25
SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
arXiv:2605. 01708v3 Announce Type: replace-cross Abstract: Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase.