Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

Simon Willison
Oct 23

Introducing Kimi K3.1 Pro

Simon Willison announces the release of Kimi K3.1 Pro on the Kimi API, noting it is not yet open‑weights but full weights will be available by Nov 4 2026. The model offers a high‑speed variant 2‑3× faster at double the price, with US dollar pricing of $2.00/million input tokens and $10/million output tokens—matching GPT‑6.1 Sol and half the cost of Claude Opus 5.5. Benchmarks from Moonshot show Kimi roughly on par with Opus 5.5 and GPT‑6 Astra, scoring 55 in Artificial Analysis, ahead of several competitors.

arXiv Machine Learning
Oct 23

MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

MoEZip is a routing‑aware key‑value (KV) cache compression technique designed for sparse Mixture‑of‑Experts (MoE) language models. It introduces a Fisher‑information‑based metric to assess how sensitive expert routing is to cache perturbations, and combines this with attention weights to score contexts for eviction. Experiments demonstrate that, even with only 10% of the KV cache retained, MoEZip outperforms state‑of‑the‑art baselines by significant margins on LongBench and RULER benchmarks.

By Minsang Kim, Seungjun Baek
arXiv AI
Oct 23

DiffResearch: Accelerating Deep Research with Diffusion Language Models

DiffResearch is an open multi‑agent deep research framework that replaces the traditional autoregressive language model backbone with a diffusion language model (dLLM) across all stages of research, from planning to writing. In experiments on 100 Deep Research Bench tasks, the diffusion backbone achieves roughly six times faster end‑to‑end latency, largely due to significant speedups in planning and long‑form generation. The framework also introduces dFast, a decomposition mode that reads intermediate denoising states to generate subqueries, yielding a median 1.99× speedup without compromising report quality and achieving the best overall score on the RACE rubric. whyItMatters":"The study demonstrates that diffusion language models can dramatically reduce latency in deep research systems, enabling faster interactive use while maintaining or improving output quality."

By Alan-Barsag Gazzaev, Alexey Gavrilov, Mikhail Mozikov, Sergey Muravyov
arXiv Machine Learning
Oct 23

NN-Based Transistor Modeling for Ultra-Fast SPICE Simulation and Standard Cell Library Characterization

The paper presents neural‑network (NN) transistor models that predict drain current and charges for a 3 nm nanosheet, and applies them to standard cell library characterization. The NN models achieve sub‑0.11 % error in delay and 0.76 % in total power across a wide range of circuits, while reducing characterization time by up to eight times compared to the industry‑standard Verilog‑A BSIM‑CMG model. The study also explores how NN size affects simulation precision and provides guidance for accurate circuit simulation.

By Rodion Novkin, Hussam Amrouch
arXiv Machine Learning
Oct 23

LLM2Cert: LLM Synthesis of Certificates with SMT/Lean Verification

The paper introduces DynaCertBench, a benchmark of 105 diverse systems for certifying stability, safety, and related properties, and presents LLM2Cert, a framework that uses large language models to generate and verify candidate certificates via SMT solvers or Lean. LLM2Cert achieves a 77.1% verification rate on the benchmark, improving by 17.1 percentage points over a single-shot approach, and successfully verifies 22 certificates that All-SMT could not resolve. The authors provide code and data on GitHub for reproducibility.

By Zhipeng Cao, Peiyang Song, Nikolaus Vertovec, Weiru Cui, Zhihao Yang, Mingkang Liu, Luxia Lin, Bai Xue, Alessandro Abate
arXiv Computation and Language
Oct 23

WaxalNet: Benchmarking and Adapting Speech Recognition for Spontaneous Speech in 19 African Languages

The paper introduces WaxalNet, a benchmark and adaptation study for spontaneous speech recognition across 19 African languages using the WAXAL corpus. It evaluates zero‑shot and fine‑tuned performance of several multilingual foundation models (Whisper, MMS, Omnilingual ASR, SeamlessM4T‑v2), showing that fine‑tuning can substantially reduce word error rates and that LLM decoders generally outperform CTC decoders when the encoder is shared. The authors also demonstrate that a single multilingual model can serve all languages with only a modest WER penalty and provide a native‑speaker audit of failure modes.

By Victor Tolulope Olufemi, Oreoluwa Babatunde, Ramsey Njema, Bolarinwa Gbotemi, Wanchi Lucia Yen, John Uzodinma, Sunday Ajayi, Oluwademilade Williams, Kausar Moshood, Innocent Elendu Anyaele, Akebert Arefaine, Candace Hunzwi, Wongel Dawit Daniel, Emmilly Namuganga, Cleophas Kadima, Athanase Bahizire, Onitsiky Ranaivoson, Emmanuel Aaron, Nicholaus Ladislaus, Idris Muhammed, Jonathan Enoch Simenya, Martin Koome, Matewos Tegete Endaylalu, Peter Ifeoluwa Adeyemo, Hondi Prisca Birindwa, Ukachi Agnes Eze-Mbey, Yacoba Oduro-Yeboah, Pericles Adjovi, Mikel K. Ngueajio, Toluwani Aremu, Prasenjit Mitra