← Back to all news

Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning
Oct 8

Geological text descriptions in ill-posed inverse problems: insights from learned hydraulic-conductivity inversion

The paper investigates how geological text descriptions can aid hydraulic‑conductivity inversion, a problem that remains ill‑posed even with complete head observations. Using a synthetic Darcy‑flow benchmark, the authors show that after training with sparse observations, incorporating textual descriptions can improve reconstruction when structural ambiguity persists, though the benefit varies with observation density, flow direction, and specific content. They find that editing the text shifts reconstructions toward the described geometry, but they do not demonstrate a clear advantage of text over equivalent numerical inputs, and they suggest that these insights could inform joint measurement and knowledge acquisition strategies.

By Taiga Saito, Yu Otake, Daijiro Mizutani, Sopheakpolin Mom
benchmarks
More like this →
arXiv Computation and Language
Oct 8

PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs

PHRBench is a new benchmark that evaluates how large language models handle hallucinated information during multi-stage reasoning. It tests 18 models across four domains using 4,820 controlled instances, measuring Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction to determine if a model can correct hallucinated premises and still reach the correct answer. The study finds that successful recovery is rare, often requires frequent belief updates, and that features of the hallucinated prompt can predict recovery with an AUROC of 0.847.

By Linghao Meng, Feng He, Xuan Yang, Junyuan Mao, Pinze Ren, Deqing Mu, Hesen Yang, Qiankun Li
llmsbenchmarkssafety
More like this →
arXiv Computation and Language
Oct 8

EmphTTS: an emphasis-control TTS with reinforcement learning

EmphTTS is a non‑autoregressive text‑to‑speech system that uses Group Relative Policy Optimization (GRPO) to train its duration predictor with an emphasis localization reward, allowing direct word‑level emphasis control. The system outperforms existing baselines in both objective emphasis evaluation and subjective preference tests, and ablation studies confirm that GRPO improves emphasis realization while reducing mismatch between the duration predictor and the TTS model.

By Zirui Li, Rech Silas, Lauri Juvela, Tom Backstrom, Mikko Kurimo
reinforcement-learningfine-tuningmultimodalbenchmarks
More like this →
arXiv Machine Learning
Oct 8

Transmission Factors for Lossy Compression of PDE Training Data: Measuring What Reaches a Trained Operator

The paper discusses how operator‑learning benchmarks are stored as full‑precision arrays that have grown to terabyte scale, and curators must decide how coarsely to compress them. It argues that the usual tolerance on reconstruction error of the stored field is measured in the wrong place, because what matters is the accuracy of a model trained on the compressed copy, not the raw field error. The authors propose a new metric: push a compressed field through a model trained at full precision and compare its output to the output on the clean field, which captures the portion of codec error that actually reaches the model’s output. This metric spans more than two orders of magnitude across six PDE families and correctly orders downstream cost where field error does not, reversing many cross‑dataset comparisons.

By Huy Hoang Le
benchmarks
More like this →
arXiv Machine Learning
Oct 8

An Invariant Tangent-Angle Descriptor and a Band U-Net for 2D Fragment Adjacency Prediction

arXiv:2610.09459v1 Announce Type: cross Abstract: This paper addresses the prediction of adjacency between pairs of 2D fragments based on their contours. We improved the two-stage architecture propos...

By Guillaume Brouillette (Universit\'e du Qu\'ebec \`a Trois-Rivi\`eres, Trois-Rivi\`eres, Canada), Alain Goupil (Universit\'e du Qu\'ebec \`a Trois-Rivi\`eres, Trois-Rivi\`eres, Canada), Pierre-Olivier Paris\'e (Universit\'e du Qu\'ebec \`a Trois-Rivi\`eres, Trois-Rivi\`eres, Canada), Fadel Tour\'e (Universit\'e du Qu\'ebec \`a Trois-Rivi\`eres, Trois-Rivi\`eres, Canada)
benchmarks
More like this →