How to train a new language model from scratch using Transformers and Tokenizers

Training a language model with 🤗 Transformers using TensorFlow and TPUs

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

arXiv:2512. 20757v2 Announce Type: replace-cross Abstract: Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs).

Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.

How to generate text: using different decoding methods for language generation with Transformers

Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Train and Fine-Tune Sentence Transformers Models