Making thousands of open LLMs bloom in the Vertex AI Model Garden

Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate

Optimizing your LLM in production

Optimization story: Bloom inference

How To Build Your Own LLM Runtime From Scratch

If you have ever wanted to actually build an LLM inference runtime yourself — pack your own weights, own every barrier, capture your own CUDA graphs — this is what that journey looks like on an H100. A step-by-step tour of a small runtime called annotated-llm-runtime, and the three bugs that produced most of the annotations.

Understanding and Coding the KV Cache in LLMs from Scratch

KV caches are one of the most critical techniques for efficient inference in LLMs in production.

Apertus LLM Family Expansion via Distillation and Quantization

arXiv:2605. 29128v2 Announce Type: replace Abstract: The wide adoption of LLMs has led to their use in great variety of applications and scenarios, such as chatbot assistants and data annotation, creating the need for the models to satisfy certain budget and hardware constraints.