ibUMAP introduces a coherent, field‑based alternative to UMAP’s stochastic negative sampling, evaluating attraction and repulsion from a shared embedding snapshot synchronously. By representing the degree‑weighted repulsive field with three scalar moments and using an interpolation‑based FFT scheme, it avoids explicit all‑pairs computations while maintaining efficient CPU and GPU performance. Benchmarks show median speedups of 3.29× (CPU) and 1.44× (GPU) over existing implementations, with improved run‑to‑run stability and controlled fidelity trade‑offs.
arXiv:2605. 01910v2 Announce Type: replace-cross Abstract: Autoregressive decoding becomes bandwidth-limited at long contexts, as generating each token requires reading all $n_k$ key and value vectors from KV cache.
arXiv:2506. 01969v3 Announce Type: replace-cross Abstract: Efficient inference of Multi-Head Latent Attention (MLA) is challenged by deploying the DeepSeek-R1 671B model on a single Multi-GPU server.
arXiv:2609.05760v1 Announce Type: cross Abstract: We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU enviro...
The paper introduces a MeanField surrogate model for predicting performance of concurrent heterogeneous AI inference workloads on shared GPUs, reducing profiling complexity from combinatorial to linear in the number of models. Experiments with up to six models show high accuracy (R²≈0.96) and efficient integration into a genetic algorithm scheduler, achieving near-exhaustive search performance with minimal runtime overhead.
The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.
Orthrus is a serving system that performs heterogeneous batching of embedding and generative models within a single inference loop. It uses chunked embedding with incremental pooling and workload‑aware batch composition to unify conflicting computational patterns. Experiments on four A100 GPUs show that Orthrus improves throughput by 1.28×–4.52× and reduces p99 latency by up to 55.8% compared to baseline deployments.
The paper investigates how well common large‑language‑model (LLM) inference optimizations perform on Apple Silicon compared to server‑class NVIDIA GPUs. Four optimization families—KV‑cache pruning, speculative decoding, ported GPU kernels, and post‑training quantization—are evaluated on four Apple machines under a single‑request decode workload, with matched RTX 4090 comparisons where possible. Results show that KV‑cache pruning can boost Apple throughput up to 1.75×, while speculative decoding actually reduces throughput on Apple; CUDA kernel designs transfer only partially, and quantization gains vary by machine, indicating that speedups depend on the specific technique, runtime, hardware, and workload regime.
The paper introduces a composite metric for quantizing small language models that balances information retention and throughput gains, using a normalized SQNR-based coefficient and roofline-based latency analysis. Profiling Gemma 3 1B shows that Feed‑Forward Network blocks and the embedding matrix are prime candidates for acceleration, with the metric enabling tuning of speed‑quality trade‑offs without actual execution. The authors demonstrate that their estimates predict accelerated speedup within about 4% error and allocate resources more effectively than evolutionary search or Shapley‑value methods.
arXiv:2607.01455v4 Announce Type: replace-cross Abstract: Language models learn continuous programs over discrete symbols, with the embedding table and LM-head acting as the read/write interface betw...
arXiv:2607. 14541v1 Announce Type: new Abstract: Existing GPU kernel generation benchmarks draw problems from synthetic or curated sources that diverge from deployed workloads.
Configuring distributed LLM training across hundreds of GPUs requires jointly optimizing tensor, pipeline, expert, context, and data parallelism alongside micro-batch size, recomputation, sharding, an...
arXiv:2607. 01455v1 Announce Type: cross Abstract: Language models learn continuous programs over discrete symbols, with the embedding table and LM-head acting as the read/write interface between them.