How to train a new language model from scratch using Transformers and Tokenizers

Training a language model with 🤗 Transformers using TensorFlow and TPUs

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

arXiv:2512. 20757v2 Announce Type: replace-cross Abstract: Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs).

Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.

How to generate text: using different decoding methods for language generation with Transformers

Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Train and Fine-Tune Sentence Transformers Models

The Reformer - Pushing the limits of language modeling

How to train a Language Model with Megatron-LM

Efficient training of language models to fill in the middle

Evaluating large language models trained on code

TokEval: A Tokenizer Evaluation Suite

Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance.

Prototype Transformer: Towards Language Model Architectures Interpretable by Design

arXiv:2602. 11852v2 Announce Type: replace Abstract: While state-of-the-art language models (LMs) surpass most humans in certain domains, their reasoning remains largely opaque, reducing trust and increasing the risk of deception and hallucination.