Towards Data Science
Jun 25
3 Agents. 3 LLMs. 1 Aging GPU: Engineering Parallel Inference on Bare Metal
Beat the 8GB VRAM limit. Learn how to run three different LLMs on a single 8GB GPU using C++ layer multiplexing and admission control.
I Built a C++ Backend So My GPU Would Stop Eating Air
A comprehensive guide to optimizing LLM inference by eliminating padding overhead with hardware-aware sequence packing. The post I Built a C++ Backend So My GPU Would Stop Eating Air appeared first on Towards Data Science .
Related stories
Towards Data Science
Jun 19
GPU-Resident Top-K for Agentic RAG: I Built a CUDA Kernel So My Retrieval Step Would Stop Bouncing Off the GPU
The PCIe transfer latency is silently bottlenecking your agentic inference. Here is how building a custom device-resident vector search kernel bypasses the CPU to unlock deterministic microsecond tail latencies.
arXiv AI
Jun 15
UltraSketchLLM: Sub-1-Bit LLM Compression via Sketch and Hardware-Friendly Operators
arXiv:2506. 17255v2 Announce Type: replace-cross Abstract: Large language models (LLMs) require larger GPU memory size these days, necessitating efficient and extreme weight compression methods.
Towards Data Science
Jul 22
How To Build Your Own LLM Runtime From Scratch
If you have ever wanted to actually build an LLM inference runtime yourself — pack your own weights, own every barrier, capture your own CUDA graphs — this is what that journey looks like on an H100. A step-by-step tour of a small runtime called annotated-llm-runtime, and the three bugs that produced most of the annotations.
arXiv AI
Jul 1
An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU
arXiv:2603. 16428v2 Announce Type: replace-cross Abstract: Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs.
Towards Data Science
Jul 21
How Much of a Data Science Workflow Can Run on a GPU Today? Part 1: Accelerating Data Preparation
Exploring GPU acceleration with cuDF, cudf. pandas, and the Polars GPU Engine The post How Much of a Data Science Workflow Can Run on a GPU Today?
Towards Data Science
Jun 9
Prefill Once, Fan Out: KV Snapshot Sharing for Multi-Agent LLM Pipelines
Stop re-computing the same context. Learn how to build a C++ runtime with copy-on-fork KV snapshots to eliminate redundant LLM prefills in multi-agent pipelines.
arXiv Machine Learning
Jul 30
LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving
arXiv:2607. 26491v1 Announce Type: cross Abstract: The energy consumption of Large Language Model (LLM) serving is becoming a major system challenge as deployment scales, driven by hardware power and thermal constraints and rising electricity costs.
arXiv Machine Learning
Jun 5
CuTeGen: An LLM-Based Agentic Framework for Generation and Optimization of High-Performance GPU Kernels using CuTe
arXiv:2604. 01489v2 Announce Type: replace Abstract: High-performance GPU kernels are critical to modern machine learning systems, yet developing them remains a manual, expert-driven process.
arXiv Machine Learning
Aug 6
SparseDitto: Customizing GPU Kernels for Different Sparsity Patterns with LLM-Based Agentic System
arXiv:2608. 05033v1 Announce Type: cross Abstract: Sparse matrix kernels are fundamental to scientific computing, graph analytics, and machine learning.
arXiv Machine Learning
Jun 9
Towards Automated Kernel Generation in the Era of LLMs
arXiv:2601. 15727v3 Announce Type: replace Abstract: The performance of modern AI systems is fundamentally constrained by the quality of their underlying GPU kernels, which translate high-level algorithmic semantics into low-level hardware operations.
arXiv AI
Aug 11
LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
arXiv:2608. 08382v1 Announce Type: new Abstract: As LLM inference shifts to multi-tenant GPU clusters, co-batching improves throughput but obscures per-tenant usage and limits control.