The Reformer - Pushing the limits of language modeling

Understanding the capabilities, limitations, and societal impact of large language models

surprisal is Not a Theory

The article argues that Surprisal Theory, often presented as a computational-level explanation, is not a theory in its own right. It contends that using large language model (LLM) surprisals without considering the underlying representational and algorithmic choices obscures the theory’s commitments. The authors demonstrate through three analyses that algorithm and architecture significantly influence language model probabilities, urging researchers to reassess treating LLM surprisals as interchangeable.

Aligning language models to follow instructions

Scaling laws for neural language models

How to train a new language model from scratch using Transformers and Tokenizers

Red-Teaming Large Language Models