Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

Simon Willison
Sep 29

Quoting Anthropic Frontier Red Team

The article reports that on a set of 100 randomly selected tasks from an internal Binary Exploitation benchmark, GLM‑5.3 achieved full control‑flow hijacks in 4% of the trials, while Claude Mythos Preview did so in 6%. Both models outperform earlier versions such as Claude Opus 4.6 and GLM‑5.2, which succeeded in none of the trials. This indicates that a significant threshold in adversarial exploitation capabilities has been crossed by the newer models.

Towards Data Science
Sep 29

I Compacted 1,000 Apache Iceberg Files Into 6. Here’s What Happened to Query Performance.

The article reports on a benchmark that reduced 1,000 Apache Iceberg files to just six, then measured how this consolidation affected query performance across three SQL workloads. It details the methodology and results of the experiment, highlighting changes in execution speed and resource usage. The findings illustrate the trade‑offs between file count and query efficiency in large data systems.

By Thomas Reid