Simon Willison

GPT‑6 Astra

GPT‑6 Astra is a new OpenAI model rolling out today to a limited set of organizations and soon to all ChatGPT Plus, Pro, Business, Enterprise users, and via the OpenAI API and AWS. It is priced at $10/million input and $50/million output, matching Claude Fable 5/5.1, and outperforms Fable on most OpenAI self‑reported benchmarks, achieving 99.9% on the ARC‑AGI 3 benchmark with a custom Provider Adapter harness. Astra excels in security tasks—scoring 100% on ExploitBench, 42.4% on ExploitGym, and 99.2% on SRE‑Bench—and handles long context well, hitting 100% on OpenAI’s eight‑needle benchmark at 256K–512K tokens and 96.3% at 512K–1M tokens, though it remains behind Fable on the Intelligence Index and Meta’s Muse Spark 1.3.

Simon Willison
Sep 1

Claude Fable 5.1 made me a really nice animated pelican

The article discusses Anthropic’s Claude Fable 5.1 release, highlighting its claimed improvements in coding, knowledge work, and problem‑solving, particularly a 52.6% score on the new Terminal‑Bench‑Science 0.1 benchmark. The author examines the model’s performance on the pelican benchmark, noting that Fable 5.1’s five reasoning levels (low, medium, high, xhigh, max) sometimes skip reasoning entirely for certain prompts, as evidenced by token counts and cost metrics. The piece provides detailed transcript data for each reasoning level when generating an SVG of a pelican riding a bicycle.

arXiv Machine Learning
Jun 11

Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

arXiv:2606. 12344v1 Announce Type: new Abstract: General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring.

By Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu, Yuchuan Tian, Wei He, Hang Zhou, Jianyuan Guo, Hailin Hu, Lin Ma, Chao Xu, Guohao Dai, Lixue Xia, Yunchao Wei, Yunhe Wang, Yu Wang
arXiv AI
Jul 16

Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs

arXiv:2607. 13080v1 Announce Type: cross Abstract: Autonomous coding agents force engineering organizations to choose between API-based frontier models -- strong reasoning at high token cost -- and on-premise quantized open-weights models, which promise low-marginal-cost scaling and data sovereignty at some loss of reasoning fidelity.

By Sheng-Wei Peng, Yi-Hsun Lin, Yi-Pei Lee
Simon Willison
2d ago

Introducing Kimi K3.1 Pro

Simon Willison announces the release of Kimi K3.1 Pro on the Kimi API, noting it is not yet open‑weights but full weights will be available by Nov 4 2026. The model offers a high‑speed variant 2‑3× faster at double the price, with US dollar pricing of $2.00/million input tokens and $10/million output tokens—matching GPT‑6.1 Sol and half the cost of Claude Opus 5.5. Benchmarks from Moonshot show Kimi roughly on par with Opus 5.5 and GPT‑6 Astra, scoring 55 in Artificial Analysis, ahead of several competitors.

Hugging Face Trending Papers
Jun 10

Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

General-purpose agents such as OpenClaw are increasingly used as autonomous tool users, but their coding ability is difficult to measure under SWE-bench: a generic agent does not by itself satisfy the clean Docker workspace, patch, and prediction contract required for scoring. We introduce Claw-SWE-Bench, a multilingual SWE-bench-style benchmark and adapter protocol that makes heterogeneous agent harnesses, or claws, comparable under fair settings including a fixed prompt, runtime budget, workspace contract, patch extraction procedure, and evaluator.

OpenAI Blog
Sep 29

Introducing GPT-6.1 Sol

OpenAI introduces GPT‑6.1 Sol, a new model described as near‑Astra intelligence for coding, computer use, and professional work. It offers these capabilities at one‑fifth of Astra’s standard API input and output token prices, implying a more cost‑effective solution for developers and businesses.

Simon Willison
Aug 23

Anthropic’s best AI model struggles to attract users as cheaper tools thrive

Anthropic’s top AI model is struggling to attract users even as cheaper alternatives thrive. The company’s July revenue is projected at $65 bn, up from $47 bn in May, and it expects Q3 profitability while boasting 6,000 high‑spending customers. In contrast, OpenAI’s revenue has risen 35 % this quarter, spurred by GPT‑5.6, and a Ramp AI index shows Anthropic’s newer models (e.g., Fable) are less popular than older ones like Opus 4.8.

Simon Willison
Aug 20

ChatGPT search now uses the site:operator at scale

ChatGPT search now uses the site:operator at scale Promptwatch is part of the emerging "GEO" space, for Generative Engine Optimization - the chatbot version of SEO, where companies offer tools and consulting to help your site increase its presence in replies to prompts inside tools like ChatGPT. The Promptwatch product uses automation to track responses to prompts across end-user chat products like ChatGPT, Claude, and Gemini.

arXiv AI
Oct 2

Finding the Right Fit: Model-Harness Interactions across Agent Tasks

The study evaluates 66 combinations of four configurable harnesses and five language models across three benchmark suites, revealing that model rankings and best harnesses vary significantly with the task. Notably, the openJiuwen harness consistently yields the highest scores for the Kimi model, while GPT performs best with the lean PI scaffold on Terminal‑Bench 4. The findings demonstrate that a model’s performance is highly dependent on the harness it is paired with, and that higher cost or a vendor’s own harness does not guarantee superior results.

By Yixuan Li, Yiyun Zhou, Yao Long Teng, Fuchao Yang, Yanchen Deng, Zhiyi Lyu, Xuyu Dong, Feng Chen, Bo An