Large language models

Hugging Face Reads, Feb. 2021 - Long-range Transformers

Retrieval Augmented Generation with Huggingface Transformers and Ray

Understanding the capabilities, limitations, and societal impact of large language models

Faster TensorFlow models in Hugging Face Transformers

Scaling Kubernetes to 7,500 nodes

We’ve scaled Kubernetes clusters to 7,500 nodes, producing a scalable infrastructure for large models like GPT-3, CLIP, and DALL·E, but also for rapid small-scale iterative research such as Scaling Laws for Neural Language Models.

How we sped up transformer inference 100x for 🤗 API customers

CLIP: Connecting text and images

We’re introducing a neural network called CLIP which efficiently learns visual concepts from natural language supervision. CLIP can be applied to any visual classification benchmark by simply providing the names of the visual categories to be recognized, similar to the “zero-shot” capabilities of GPT-2 and GPT-3.

Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

Porting fairseq wmt19 translation system to transformers

Hyperparameter Search with Transformers and Ray Tune

Transformer-based Encoder-Decoder Models

OpenAI licenses GPT-3 technology to Microsoft

OpenAI has agreed to license GPT-3 to Microsoft for their own products and services.

Block Sparse Matrices for Smaller and Faster Language Models

Learning to summarize with human feedback

We’ve applied reinforcement learning from human feedback to train language models that are better at summarization.

Image GPT

We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. By establishing a correlation between sample quality and image classification accuracy, we show that our best generative model also contains features competitive with top convolutional nets in the unsupervised setting.

Language models are few-shot learners

How to generate text: using different decoding methods for language generation with Transformers

How to train a new language model from scratch using Transformers and Tokenizers

Scaling laws for neural language models

Deep double descent

We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization.