OpenAI Blog
Jan 18, 2018
Scaling Kubernetes to 7,500 nodes
We’ve scaled Kubernetes clusters to 7,500 nodes, producing a scalable infrastructure for large models like GPT-3, CLIP, and DALL·E, but also for rapid small-scale iterative research such as Scaling Laws for Neural Language Models.
Related stories
OpenAI Blog
Jan 23, 2020
Scaling laws for neural language models
arXiv AI
Jul 20
Kolmogorov--Arnold Networks for Small Language Models
arXiv:2607. 15525v1 Announce Type: cross Abstract: Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks.
arXiv AI
Jul 29
Does Runtime Topology Context Improve LLM-Generated Kubernetes Security Patches?
arXiv:2607. 25995v1 Announce Type: cross Abstract: Kubernetes is central to the cloud-native ecosystem, orchestrating containerised workloads.
OpenAI Blog
Jun 9, 2022
Techniques for training large neural networks
Large neural networks are at the core of many recent advances in AI, but training them is a difficult engineering and research challenge which requires orchestrating a cluster of GPUs to perform a single synchronized calculation.
arXiv AI
Jun 19
Towards Engineering Scaling Laws with Pretraining Data Composition
arXiv:2606. 19781v1 Announce Type: cross Abstract: Neural scaling laws describe how model performance improves as a power law in compute, model size, and dataset size.
arXiv Machine Learning
Jun 11
Scaling Laws of Global Weather Models
arXiv:2602. 22962v2 Announce Type: replace Abstract: Data-driven models are revolutionizing weather forecasting.
arXiv Machine Learning
Aug 4
Kilobyte Models: Neural Networks as a Seed and a Quantized Latent
arXiv:2608. 00860v1 Announce Type: new Abstract: The cost of storing and transmitting a trained neural network scales with its parameter count, a bottleneck for over-the-air updates, on-device libraries, and other bandwidth-bound deployments.
arXiv Machine Learning
Jul 3
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
arXiv:2607. 01487v1 Announce Type: new Abstract: We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called three-term law).
arXiv AI
Jun 18
FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs
arXiv:2606. 19025v1 Announce Type: cross Abstract: Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators.
arXiv AI
Jun 17
KANLib -- An Modular, Extensible and Fast Kolmogorov-Arnold Network Implementation
arXiv:2606. 17927v1 Announce Type: cross Abstract: Kolmogorov-Arnold Networks (KANs) have recently emerged as a promising alternative to traditional multilayer perceptrons by replacing linear weights with learnable univariate functions.
arXiv Machine Learning
Jul 16
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
arXiv:2607. 13332v1 Announce Type: new Abstract: Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel (MP) techniques presume homogeneous accelerators, high-speed interconnects, and a single orchestrating entity.