Accelerated Inference with Optimum and Transformers Pipelines

How we sped up transformer inference 100x for 🤗 API customers

Sparse Layers are Critical to Scaling Looped Language Models

arXiv:2605. 09165v2 Announce Type: replace Abstract: Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries.

Recirculation

arXiv:2608. 17981v1 Announce Type: new Abstract: We describe an inference-time architectural enhancement for off-the-shelf foundation models that markedly reduces perplexity and boosts accuracy across generation and reasoning tasks.

ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour

arXiv:2607. 23478v1 Announce Type: cross Abstract: Fully homomorphic encryption (FHE) provides strong cryptographic guarantees for private inference, but deploying transformer models under FHE remains prohibitively expensive.

Up to 3.2x Faster Inference with LFM2.5-DSpark

Scaling up BERT-like model Inference on modern CPU - Part 2