Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

Simon Willison
Sep 27

2026 in LLMs (so far)

Simon Willison delivered a closing keynote at the WeAreDevelopers World Congress North America, outlining the key developments in large language models for 2026. He highlighted the release of Claude Opus 4.5 and GPT‑5.1 in November 2025, noting that their coding agents became reliable enough for daily use. Willison also discussed early benchmarks, such as generating an SVG of a pelican riding a bicycle, and mentioned a new GitHub repository called "Warelay" that emerged during the year.

arXiv Computation and Language
Sep 25

Personalized Korean Lipreading as Visual Speech Recognition: Transfer, Census and Adaptation on OLKAVS

The paper introduces a personalized Korean visual speech recognition system that uses a video-only Conformer model initialized from English-trained weights, achieving a character error rate (CER) of 9.95–12.19% on the OLKAVS nine-camera corpus and 19.00–21.52% on unseen wording. Individual speaker CER varies widely (1.0–52.2%), with seen wording reducing errors by 7.0–9.0 points and professional or spontaneous speech increasing errors by 8.5–12.7 points. A low‑rank adapter, comprising only 4.6% of the model parameters and trained on 4–29 minutes of a user’s frontal video, reduces high‑error speakers’ CER by 2.13–3.58 points, transfers across all cameras without loss, and retains 85% of full fine‑tuning benefits at 12% of its cost; cameras above the mouth plane add a constant offset of about six CER points that can be mitigated by training on all views.

By Se Un Park, Hakjun Kim, Taehoon Roh, Junyoung Park
arXiv Computer Vision
Sep 25

GeoBlur: Epipolar Geometry Estimation from a Single Motion-Blurred Image

GeoBlur is a framework that estimates the fundamental matrix and relative camera pose from a single motion‑blurred image by exploiting blur artifacts as motion cues. It predicts visual correspondences between two time instances within the exposure window and solves the single‑frame epipolar geometry problem, yielding a fundamental matrix unique up to transposition due to time‑direction ambiguity. The method shows improved performance on synthetic and hybrid benchmarks and remains competitive on real motion‑blur data, also enabling downstream single‑frame motion segmentation.

By Bao-Long Tran, Cuong Le, Fredrik Viksten, Per-Erik Forss\'en
arXiv Computation and Language
Sep 25

SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data

SemMSA introduces a latent semantic‑aided framework for multimodal sentiment analysis that leverages large language models to generate rich sentiment‑relevant semantics. The method employs Cross‑modal Semantic Refinement (CSR) to fuse visual, acoustic, and language features in a frozen LLM embedding space, and Cross‑modal Spectral Alignment (CSA) to align these refined semantics with all modalities via spectral enhancement of kernel Gram matrices. Experiments on SIMS, MOSI, and MOSEI benchmarks show that SemMSA achieves state‑of‑the‑art performance.

By Wenhao Li, Zhibin Wu, Chong Xiao, Qiangchang Wang
arXiv Computer Vision
Sep 25

OmniFabric: Coherent UV Space Texture Synthesis for 3D Garment Reconstruction

OmniFabric is a new method for creating production‑ready 3D garment assets from a single image. It generates globally coherent texture maps directly in the 2D sewing pattern (UV) space, using a coarse initialization from Vision‑Language Models and refining it with a diffusion transformer conditioned on 3D positional features. The approach removes distortion and baked‑in artifacts, producing photorealistic 3D garments with high‑quality textures that outperform current state‑of‑the‑art baselines.

By Ding-Jiun Huang, Yuanhao Wang, Cheng Zhang, Hugo Bertiche, Alexandru-Eugen Ichim, Thabo Beeler, Fernando De la Torre
arXiv AI
Sep 25

TWIST: A Proposed Benchmark for Intervention Quality in Conversational Memory, with a Human-Validated Draft-Alignment

TWIST is a new benchmark suite designed to evaluate the quality of interventions in conversational memory systems, focusing on how well these systems act correctly at belief change points. It includes four tracks—tension detection, draft vetting, belief-consistent answering, and sensitive recall governance—each paired with hard-negative controls to prevent gaming. The benchmark is rigorously validated through double annotation, adjudication, and a separability audit, revealing that current models face trade-offs between contradiction recall and false-positive rates.

By Subrat Panda
arXiv AI
Sep 25

MeshHeal: Two-Timescale Self-Healing for Gray Failures in Decentralized LLM Agent Networks

MeshHeal is a fully decentralized self‑healing framework for decentralized LLM‑based multi‑agent systems that addresses gray failures—situations where an agent remains responsive but its task‑solving quality degrades. It operates on two timescales: a fast adaptive hierarchy that escalates uncertain or low‑scoring outputs to committee review and correction, and a slow peer‑relative detector that aggregates scores to distinguish persistent degradation from normal variation, triggering mandatory review and eventual exclusion of degraded agents while allowing recovered agents to rejoin. MeshHeal’s evaluation, using Model‑Backed MAS Evaluation, shows it achieves higher degraded‑phase accuracy (0.839) on BBH, MATH, and MMLU‑Pro with fewer tokens per task compared to the baseline Symphony.

By Keru Chen, Sen Lin, Yingbin Liang, Nathaniel D. Bastian, Shaofeng Zou
arXiv AI
Sep 25

Augur: A Synthetic Decision Lab for Rehearsing Reactions to Product and Policy Changes

Augur is a synthetic decision laboratory that simulates how users will react to product and policy changes before they are released. It constructs a typed knowledge graph from change documents, populates a persona market, runs simulations, and produces an auditable decision memo recommending one of five actions. Using a dataset of 50 real episodes (Gold‑50), the authors evaluate the system’s five‑way release verdicts and find that evaluation design, rather than model capability, largely drives performance differences among frontier and open‑weight models.

By Rahul Khedar, Mayank Malhotra, Avinash Karn
arXiv AI
Sep 25

GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI

GRASP is a multi-stage planning framework that improves the reliability of large language models on complex tasks. It separates planning into three specialized modules—GenPlan for global macro-guidelines, RevPlan for exploring localized strategies, and VerPlan for multi-criteria evaluation—allowing context isolation and strict macro-regularization. Experiments show GRASP outperforms direct LLM planners by significant margins on datasets such as Natural Plan Calendar Scheduling, ZebraLogic, and SciBench Math, and it mitigates performance collapse in multi-task and dual-task settings.

By Arunabh Srivastava (Amir), Mohammad A. (Amir), Khojastepour, Srimat Chakradhar, Sennur Ulukus
arXiv Computation and Language
Sep 25

Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks

The paper investigates how well large language models (LLMs) can handle character attacks—ad hominem arguments—in political debates. By analyzing natural political dialogues and comparing LLM-generated responses to a corpus of U.S. presidential debates, the study finds that most LLMs favor logical defenses and rarely use ethos-based counterattacks. The authors suggest that safety fine‑tuning limits LLMs’ strategic options, preventing them from fully engaging in realistic political discourse.

By Ewelina Gajewska, Katarzyna Budzynska, Jaroslaw Chudziak
arXiv Computation and Language
Sep 25

Clinical Intent Extraction: A FHIR-Aligned Representation and the CIRCA Benchmark

The paper introduces Clinical Intent Extraction (CIE), a task that transforms fragmented clinical action annotations into complete structured records called Clinical Intent Representation (CIR). CIR decomposes each action into verb, type, coded target, timing, condition, request‑intent (aligned to HL7 FHIR) and modality, adding dimensions absent in prior datasets. By re‑expressing five heterogeneous corpora into CIR, the authors create CIRCA, a benchmark of 10,011 harmonized intents with human‑validated subsets, crosswalks, and a deterministic FHIR R4 mapper, and demonstrate that existing models perform poorly on the full task, highlighting the need for targeted development.

By Alexander Apartsin, Yehudit Aperstein
arXiv AI
Sep 25

Neuralized Multi-Wavelet Decomposition for Time Series Classification and Forecasting

The paper introduces m-WCN, an end‑to‑end deep learning framework that neuralizes multi‑wavelet decomposition to jointly extract temporal patterns and frequency components from time series. Two task‑specific architectures built on m‑WCN—TFBC for classification and FTB for forecasting—are shown to outperform baseline models on 64 UCR datasets and seven forecasting benchmarks, achieving average improvements of nearly 20% in both tasks. The approach leverages trainable convolutional operators and orthogonality constraints to produce interpretable multi‑resolution representations.

By Xiaohan Jiang, Jingyuan Wang, Jiahao Ji, Yongyao Wang, Chen Yang, Junjie Wu
arXiv AI
Sep 25

TabSieve: Explicit In-Table Evidence Selection for Tabular Prediction

TabSieve is a select‑then‑predict framework that explicitly chooses a small set of informative rows from a table as evidence before predicting a missing target. The authors build a large synthetic dataset, TabSieve‑SFT‑40K, and introduce a reinforcement learning method, TAB‑GRPO, to jointly optimize evidence selection and prediction. Experiments on 75 classification and 52 regression tables show consistent performance gains, with TabSieve improving classification by 2.92% and regression by 4.45% over the best baseline while enhancing robustness to noisy context.

By Yongyao Wang, Ziqi Miao, Lu Yang, Haonan Jia, Wenting Yan, Chen Qian, Lijun Li
arXiv Computation and Language
Sep 25

CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models

CONSISTRE is a consistency‑aware framework for document‑level relation extraction that tackles contradictions in large language model predictions. It offers two tracks: an inference‑time track that refines black‑box LLM outputs through constraint‑aware prompting, verification, and self‑reflection, and a training‑time track that distills consistency knowledge into smaller open‑source models via supervised fine‑tuning and reinforcement learning. Experiments on DocRED show both tracks outperform baselines, with the inference‑time track matching competitive F1 scores and the training‑time track narrowing the performance gap to proprietary LLMs while reducing inference cost.

By Mingxuan Sun
arXiv Computer Vision
Sep 25

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

CinematicVQA is a new benchmark for evaluating large vision‑language models on film‑grammar reasoning. It introduces the Cinematic Scene Graph, a structured representation linking filming techniques to perceptual effects and narrative functions, and tests models on tasks beyond low‑level technique recognition. The study finds a semantic gap where models excel at describing visuals but struggle to identify underlying techniques, and shows that fine‑tuning improves performance on narrative function and multi‑hop reasoning.

By Shuo Xing, Pooja Verlani, Balu Adsumilli, Zhengzhong Tu
arXiv AI
Sep 25

Revalidation Beats Stateful Routing for Scientific Surrogates Under Distribution Shift

The study introduces RegimeShift‑Surrogates, a streaming benchmark that tests surrogate models across eight tasks and multiple regimes. It compares revalidation—choosing the model with lowest current‑window validation loss—to stateful adaptive controllers and finds that revalidation consistently outperforms stateful methods, achieving lower mean log regret in most task‑scenario combinations. The results suggest that fresh validation evidence is more valuable than carrying over past evidence when dealing with distribution shifts.

By Harshil Lodhiya
arXiv AI
Sep 25

A Living Benchmark for Information Retrieval from Electronic Health Records

The paper introduces BRIE, a continuously maintainable benchmark for evaluating large language models (LLMs) in electronic health record (EHR) information retrieval. It presents a scalable framework that automatically generates question–answer pairs from longitudinal EHR notes, validated by nineteen clinicians. The benchmark allows assessment of multiple inference strategies and highlights that state‑of‑the‑art LLMs often miss clinically important information, especially when synthesis across documents is required.

By Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani, Philip Chung, Kevin R Keet, Kameron C. Black, Andrea T. Fisher, Sarita Khemani, Jerry Liu, Stephen Ma, Saloni K. Maharaj, Rita M. Pandya, Eduardo Perez-Guerrero, Priyanka Pillai, Lisa Shieh, David J. H. Wu, James Xie, James C. McAvoy, Teresa Nguyen, Jessica Tran, Lucy Yin, Bridget Lin, Alison Callahan, Jason A. Fries, Nigam H. Shah, Emily Alsentzer
arXiv Machine Learning
Sep 25

Task-Resolved Fisher Spectroscopy for Quantum Reservoir Computing

The paper introduces task‑resolved Fisher spectroscopy for quantum reservoir computing, defining orthonormal score coordinates from prediction targets that reweight labeled histories to produce an affine family of reservoir states and measurement outcomes. It establishes a Fisher‑information hierarchy linking state quantum Fisher information, measurement record Fisher information, and moment matrices up to many‑body order, providing a quadratic form that equals the stationary capacity of the optimal linear readout. The method requires only stationary labeled records and measured outcomes, enabling predictions of held‑out capacities, necessary feature order, and measurement‑budget dependence, and demonstrates how optimizing local measurement axes can recover hidden task information in a five‑spin open reservoir.

By Yang Peng
arXiv Computation and Language
Sep 25

Do not be greedy, Think Twice: Sampling and Selection for Document-level Information Extraction

The paper introduces ThinkTwice, a framework that uses sampling and selection to improve document‑level information extraction. Instead of relying on greedy decoding, the method generates multiple candidate templates and selects the best one via an unsupervised agreement‑based approach or a supervised reward model. It also proposes a rejection‑sampling technique to create silver training data for reasoning traces, and experimental results show that ThinkTwice consistently outperforms greedy baselines and the current supervised state‑of‑the‑art.

By Mikel Zubillaga, Oscar Sainz, Oier Lopez de Lacalle, Eneko Agirre