Introducing Optimum: The Optimization Toolkit for Transformers at Scale

Native-speed vLLM transformers modeling backend

Hugging Face and Graphcore partner for IPU-optimized Transformers

LiFT: Local Search via Linear Programming for Overfitting-Controlled Transformers

arXiv:2606. 16243v1 Announce Type: new Abstract: This paper proposes a Linear Programming (LP)-based local search framework for fine-tuning pretrained transformer models with explicit control against overfitting.

Stability of Transformers under Layer Normalization

arXiv:2510. 09904v2 Announce Type: replace-cross Abstract: Despite their widespread use, training deep Transformers can be unstable.

A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using transformers, accelerate and bitsandbytes

Mixture of Experts (MoEs) in Transformers

Convert Transformers to ONNX with Hugging Face Optimum

On the Expressive Power of Transformers

arXiv:2608. 12671v1 Announce Type: new Abstract: Multi-layer transformers form the critical component of essentially all large language models (LLMs) in use today.

Gradient Heterogeneity Complements Hessian Heterogeneity in Transformer Optimization

arXiv:2502. 00213v5 Announce Type: replace-cross Abstract: Transformers are difficult to optimize with stochastic gradient descent (SGD) and largely rely on adaptive optimizers such as Adam.

Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel

Train your first Decision Transformer

Getting Started with Hugging Face Transformers for IPUs with Optimum