OpenAI Blog

Defining and evaluating political bias in LLMs

Learn how OpenAI evaluates political bias in ChatGPT through new real-world testing methods that improve objectivity and reduce bias.

arXiv Machine Learning
Sep 3

GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models

The paper introduces GPTBIAS, a framework that uses powerful large language models like GPT‑4 to evaluate bias in other LLMs. It employs specially crafted prompts called Bias Attack Instructions to probe for bias and outputs a bias score along with detailed information such as bias types, affected demographics, keywords, reasons, and improvement suggestions. Extensive experiments demonstrate the framework’s effectiveness and usability.

By Jiaxu Zhao, Meng Fang, Shirui Pan, Wenpeng Yin, Mykola Pechenizkiy
arXiv AI
Aug 28

The BS-meter: Detecting Politics and Labour through ChatGPT's Language

The paper investigates the linguistic characteristics of ChatGPT-generated text, comparing it to 1,000 scientific publications and exploring its relation to concepts of ‘bullshit’ in political speech and workplace contexts. By applying hypothesis‑testing methods, the authors demonstrate that a statistical model of bullshit can link the artificial bullshit produced by ChatGPT to the political and workplace functions of bullshit observed in natural human language.

By Alessandro Trevisan, Harry Giddens, Sarah Dillon, Alan F. Blackwell