A comprehensive guide to optimizing LLM inference by eliminating padding overhead with hardware-aware sequence packing. The post I Built a C++ Backend So My GPU Would Stop Eating Air appeared first on Towards Data Science .
Summary generated by The Flow from the publisher's feed. The full article lives at Towards Data Science.
Beat the 8GB VRAM limit. Learn how to run three different LLMs on a single 8GB GPU using C++ layer multiplexing and admission control.
The PCIe transfer latency is silently bottlenecking your agentic inference. Here is how building a custom device-resident vector search kernel bypasses the CPU to unlock deterministic microsecond tail latencies.
arXiv:2506. 17255v2 Announce Type: replace-cross Abstract: Large language models (LLMs) require larger GPU memory size these days, necessitating efficient and extreme weight compression methods.
If you have ever wanted to actually build an LLM inference runtime yourself — pack your own weights, own every barrier, capture your own CUDA graphs — this is what that journey looks like on an H100. A step-by-step tour of a small runtime called annotated-llm-runtime, and the three bugs that produced most of the annotations.
arXiv:2603. 16428v2 Announce Type: replace-cross Abstract: Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPUs.
Exploring GPU acceleration with cuDF, cudf. pandas, and the Polars GPU Engine The post How Much of a Data Science Workflow Can Run on a GPU Today?