Rethinking how we measure AI intelligence

A shared playbook for trustworthy third party evaluations

OpenAI shares guidance on third-party AI evaluations, covering how to assess model capabilities, safeguards, and validity for frontier systems.

Game Arena: Strategic LLM Evaluation in Competitive Environments

Game Arena is an open, continuously expanding platform that evaluates large language models through competitive games, allowing head‑to‑head matchups in structured environments. Unlike static benchmarks, it prevents performance saturation by increasing gameplay difficulty as models improve. The report outlines the infrastructure and presents three pilot games—Chess, Poker, and Werewolf—covering perfect information, imperfect information, and multiplayer settings, and details evaluation metrics and competition results.

This game-playing AI is the new champ at Stratego

A new AI system has become the champion at the board game Stratego, outperforming top-ranked human players. It is more efficient than other models, and its success demonstrates advanced strategic planning capabilities. The system’s performance suggests potential applications for decision-making in military maneuvers and business negotiations.

AI Evaluation Should Work With Humans

arXiv:2608. 13577v1 Announce Type: new Abstract: This position paper argues that the dominant paradigm of AI evaluation (which focuses on superhuman autonomous performance and so implicitly targets the goal of replacing humans) is guiding AI development in the wrong direction.

Strengthening our safety ecosystem with external testing

OpenAI works with independent experts to evaluate frontier AI systems. Third-party testing strengthens safety, validates safeguards, and increases transparency in how we assess model capabilities and risks.

CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V

arXiv:2604.07733v2 Announce Type: replace Abstract: Evaluating strategic decision-making in LLM-based agents requires generative, competitive, and longitudinal environments, yet few benchmarks provid...

Artificial Intelligence Index Report 2026

arXiv:2606. 15708v1 Announce Type: new Abstract: Welcome to the ninth edition of the AI Index report.

Results of the Open Source AI Game Jam

From Atari to EVE Online: Building on 15 Years of AI Research in Games

Google DeepMind partners with game studios to prototype breakthrough AI gameplay.

Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks

The article "Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks" surveys the lack of a standard definition for AI agents and organizes this ambiguity into five dimensions: environmental interaction, learning and adaptation, autonomy, goal‑directed behavior, and temporal coherence. It reviews how each dimension has been conceptualized in prior work and compiles the metrics, benchmarks, and evaluation frameworks used to assess them. The authors also introduce the Agent Compendium, a public digital resource that extends these evaluation methods, aiming to provide a common structure for evaluating and comparing agent capabilities across AI systems.

Evaluating AI’s ability to perform scientific research tasks

OpenAI introduces FrontierScience, a benchmark testing AI reasoning in physics, chemistry, and biology to measure progress toward real scientific research.

Priorities and principles for effective third party assessments

OpenAI outlines priorities and principles for rigorous, secure, and independent third-party AI safety assessments of frontier models and safeguards.