SmolLM - blazingly fast and remarkably powerful

SmolVLM Grows Smaller – Introducing the 256M & 500M Models!

Make LLM Fine-tuning 2x faster with Unsloth and 🤗 TRL

Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard

When to use what Schatten-$p$ norm in deep learning?

arXiv:2606. 15268v1 Announce Type: new Abstract: Schatten-$\infty$ based optimizers such as Muon have shown promising empirical performance, but there remains seemingly conflicting observations regarding whether they are beneficial.

Incredibly Fast BLOOM Inference with DeepSpeed and Accelerate

LionMuon: Alternating Spectral and Sign Descent for Efficient Training

LionMuon is a new optimizer that alternates between Lion’s sign-based updates and Muon’s spectral matrix-sign updates on a fixed period P, sharing a single dual-EMA momentum buffer. This design keeps the memory footprint the same as Lion and half that of AdamW while reducing the average iteration cost compared to Muon. Experiments on 124M, 355M, and 720M models show LionMuon Pareto-dominates Muon, Lion, Signum, and AdamW across datasets and architectures, achieving lower validation loss with less compute.

AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning

arXiv:2608.23816v1 Announce Type: new Abstract: Quantized fine-tuning (QLoRA) saves memory but not time. It dequantizes every 4-bit weight on the fly, so it trains more slowly than fp16 LoRA. We pres...

Understanding LoRA Rank Trade-offs in Diffusion Model Fine-Tuning

The study investigates how the rank of Low‑Rank Adaptation (LoRA) affects diffusion model fine‑tuning on CIFAR‑10 using a DDPM U‑Net. Experiments with ranks 2, 4, 8, 16, and 32 show that moderate ranks—particularly rank 4—yield the best FID scores while keeping trainable parameters, runtime, and GPU memory low. Higher ranks offer only marginal improvements at a higher computational cost, suggesting that small‑to‑moderate ranks are efficient defaults for fixed training budgets.

20x Faster TRL Fine-tuning with RapidFire AI

Beyond LoRA: Can you beat the most popular fine-tuning technique?

How 🤗 Accelerate runs very large models thanks to PyTorch

AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis

arXiv:2606. 09682v1 Announce Type: new Abstract: AutoMegaKernel (AMK) compiles a HuggingFace Llama-family model into a single persistent cooperative CUDA kernel that runs the whole forward pass in one launch, with no per-model hand-written CUDA.