Simon Willison
Aug 17
Qwen 3.8 27B scores 52 on the Artificial Analysis Intelligence Index
Qwen 3. 8 27B scores 52 on the Artificial Analysis Intelligence Index That's the same score as GPT-5.
Quoting Ted Sanders
The article discusses how large language model (LLM) performance varies across many dimensions, noting that a single factor—model size—explains much of the variation. Larger models tend to be more reliable, cost more per token, and recall more obscure facts. It also explains OpenAI’s naming convention, where version numbers indicate release order, relatedness to other models, and the scale of improvement, using examples like Luna, Terra, Sol, and Astra.
Related stories
Simon Willison
Sep 22
Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna, and a new price war
The article reports the release of new AI models: Claude Opus 5.5 by Anthropic and GPT‑6 Sol and GPT‑6 Luna by OpenAI, noting that GPT‑6 variants are priced at half the cost of their GPT‑5.6 counterparts. It provides a detailed pricing table comparing input, cached input, and output costs across several models, highlighting how GPT‑6 Luna is among the cheapest ever offered by OpenAI. The author also comments on visual differences in model outputs, noting that GPT‑6 outputs are more muted compared to GPT‑5.6.
Simon Willison
Sep 29
GPT 6.1 Sol: Near-Astra intelligence for a fifth of the price
Simon Willison comments on GPT 6.1‑Sol, describing it as "Near‑Astra intelligence for a fifth of the price." He notes that the model’s pelican illustrations are similar to those of the GPT‑6 family and provides links to the live‑blog of the keynote and to the pelican images. The post is tagged with AI, OpenAI, generative‑AI, LLMs, and playful references to pelican‑riding‑a‑bicycle.
Simon Willison
Sep 1
Claude Fable 5.1 made me a really nice animated pelican
The article discusses Anthropic’s Claude Fable 5.1 release, highlighting its claimed improvements in coding, knowledge work, and problem‑solving, particularly a 52.6% score on the new Terminal‑Bench‑Science 0.1 benchmark. The author examines the model’s performance on the pelican benchmark, noting that Fable 5.1’s five reasoning levels (low, medium, high, xhigh, max) sometimes skip reasoning entirely for certain prompts, as evidenced by token counts and cost metrics. The piece provides detailed transcript data for each reasoning level when generating an SVG of a pelican riding a bicycle.
Simon Willison
Aug 16
Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things
Friday's big release was Qwen 3. 8 27B , an Apache 2 licensed 27B parameter vision-capable LLM from Alibaba's Qwen research lab.
Simon Willison
Sep 2
llm-gemini 0.34
The release of llm-gemini 0.34 introduces the new Gemini 3.8‑Flash model, available in low, medium, and high thinking levels, and fixes an issue where async responses failed to record the resolved model version. The update also notes that Google has released Gemini 3.8‑Flash (and a restricted 3.8 Flash Cyber version) today, with example outputs (pelicans) demonstrating the model’s performance across the different thinking levels. The author highlights Gemini Flash’s speed, low cost, and competence in generating HTML, JavaScript, and Markdown‑SVG content, citing a 13‑second, 1.8‑cent example of an HTML output.
Simon Willison
Sep 21
Jev introduces a new shape of LLM - System One, aka Decision Models
TypeSafe AI’s new model, Jev, is a ‘System One’ or decision model that takes text or semi‑structured data as input and outputs floating‑point probabilities for yes/no, choice, or score questions, rather than text. It charges only for input tokens ($0.042/million) and offers free output, making it cheaper and faster than typical LLMs. The API lets users build a state object (string, array, or key‑value pairs) and query it with multiple questions, receiving confidence scores and probability distributions for each answer.
Simon Willison
Oct 6
EmbeddingGemma 2
Simon Willison comments on EmbeddingGemma 2, noting its Apache 2.0 license and expressing preference for open‑weight models over proprietary, hosted‑only options. He argues that embedding models are often used to generate and store large numbers of vectors, and a closed model could force costly re‑embedding if the vendor discontinues service. Willison prefers a hosted solution that allows him to switch to the open‑weight version if needed.
Simon Willison
4d ago
llm 0.37
The release of llm 0.37 introduces numerous new features and fixes accumulated over two months, including new GPT-6.1 models, an openai endpoint add command, and a cheaper default model (gpt-6-luna). It enhances embedding support for varied content types and incorporates many AI‑assisted pull requests from multiple authors. The update also highlights the project’s efficient handling of AI‑generated contributions, aided by triage tools like Opus 5.5 and GPT-6 Astra.
Simon Willison
Oct 1
pwasm 0.2a0
Release: pwasm 0.2a0 pwasm is one of my folly projects - an entirely vibe-coded pure Python WebAssembly engine that I built in January during my first bout of AI mania. I hadn't touched it sin...
Simon Willison
Oct 6
Mistral Large 4
The article is a comment by Simon Willison on the Mistral Large 4 model, posted on Hacker News. He discusses the saturation of benchmarks and humorously references a benchmark involving an armadillo in fishnet tights jaywalking on Mars, comparing the performance of several large language models including Claude Opus, GPT, Gemini, and Mistral Large 4.
Simon Willison
Oct 23
Introducing Kimi K3.1 Pro
Simon Willison announces the release of Kimi K3.1 Pro on the Kimi API, noting it is not yet open‑weights but full weights will be available by Nov 4 2026. The model offers a high‑speed variant 2‑3× faster at double the price, with US dollar pricing of $2.00/million input tokens and $10/million output tokens—matching GPT‑6.1 Sol and half the cost of Claude Opus 5.5. Benchmarks from Moonshot show Kimi roughly on par with Opus 5.5 and GPT‑6 Astra, scoring 55 in Artificial Analysis, ahead of several competitors.