Large language models

Faster TensorFlow models in Hugging Face Transformers

Scaling Kubernetes to 7,500 nodes

We’ve scaled Kubernetes clusters to 7,500 nodes, producing a scalable infrastructure for large models like GPT-3, CLIP, and DALL·E, but also for rapid small-scale iterative research such as Scaling Laws for Neural Language Models.

How we sped up transformer inference 100x for 🤗 API customers

CLIP: Connecting text and images

We’re introducing a neural network called CLIP which efficiently learns visual concepts from natural language supervision. CLIP can be applied to any visual classification benchmark by simply providing the names of the visual categories to be recognized, similar to the “zero-shot” capabilities of GPT-2 and GPT-3.

Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

Porting fairseq wmt19 translation system to transformers

Hyperparameter Search with Transformers and Ray Tune

Transformer-based Encoder-Decoder Models

OpenAI licenses GPT-3 technology to Microsoft

OpenAI has agreed to license GPT-3 to Microsoft for their own products and services.

Block Sparse Matrices for Smaller and Faster Language Models

Learning to summarize with human feedback

We’ve applied reinforcement learning from human feedback to train language models that are better at summarization.

Image GPT

We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. By establishing a correlation between sample quality and image classification accuracy, we show that our best generative model also contains features competitive with top convolutional nets in the unsupervised setting.

Language models are few-shot learners

How to generate text: using different decoding methods for language generation with Transformers

How to train a new language model from scratch using Transformers and Tokenizers

Scaling laws for neural language models

Deep double descent

We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization.

GPT-2: 1.5B release

As the final model release of GPT-2’s staged release, we’re releasing the largest version (1. 5B parameters) of GPT-2 along with code and model weights to facilitate detection of outputs of GPT-2 models.

Fine-tuning GPT-2 from human preferences

We’ve fine-tuned the 774M parameter GPT-2 language model using human feedback for various tasks, successfully matching the preferences of the external human labelers, though those preferences did not always match our own. Specifically, for summarization tasks the labelers preferred sentences copied wholesale from the input (we’d only asked them to ensure accuracy), so our models learned to copy.

GPT-2: 6-month follow-up

We’re releasing the 774 million parameter GPT-2 language model after the release of our small 124M model in February, staged release of our medium 355M model in May, and subsequent research with partners and the AI community into the model’s potential for misuse and societal benefit. We’re also releasing an open-source legal agreement to make it easier for organizations to initiate model-sharing partnerships with each other, and are publishing a technical report about our experience in coordinating with the wider AI research community on publication norms.