Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning
Oct 7

PRUE: A Practical Recipe for Field Boundary Segmentation at Scale

arXiv:2603.27101v2 Announce Type: replace-cross Abstract: Large-scale maps of field boundaries are essential for agricultural monitoring tasks. Existing deep learning approaches for satellite-based f...

By Gedeon Muhawenayo, Caleb Robinson, Subash Khanal, Zhanpei Fang, Isaac Corley, Alexander Wollam, Tianyi Gao, Leonard Strnad, Ryan Avery, Lyndon Estes, Ana M. T\'arano, Nathan Jacobs, Hannah Kerner
arXiv Machine Learning
Oct 7

Independent Multi-Agent Reinforcement Learning with Counterfactual Semantic-Social World Models

The paper introduces CASTLE, a framework for fully decentralized multi‑agent reinforcement learning that uses two offline‑trained world models to provide in‑context guidance to independent agents. A Local Dynamics World Model captures each agent’s trajectory dynamics, while a Semantic‑Social World Model predicts short‑horizon task and social outcomes for candidate actions based on counterfactual simulator rollouts. During online learning, these frozen models supply prediction logits to an independent PPO policy, enabling agents to compare potential actions prospectively rather than relying solely on scalar rewards. CASTLE outperforms existing baselines on several multi‑particle benchmark tasks, achieving the highest mean final scores across 30 seeds.

By Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen
arXiv Machine Learning
Oct 7

Machine Learning for German Redispatch Forecasting under Data Delays and Temporal Distribution Shift

The study evaluates probabilistic machine‑learning models for forecasting German grid redispatch volumes under data‑delay constraints. Using 48,242 records from 2021‑2024, boosted‑tree LightGBM with rolling calibration achieved the best performance (nWIS 0.7767), outperforming autoregressive and seasonal baselines. Neural models with zero‑censored outputs performed similarly but revealed undercoverage during high‑volume events.

By Faraz Shamim (KIST Medical College and Teaching Hospital, Nepal), Faris Shamim (OTH Regensburg)
arXiv Computation and Language
Oct 7

Nord-Parl-TTS: Finnish and Swedish TTS Dataset from Parliament Speech

Nord-Parl-TTS is an open text‑to‑speech dataset for Finnish and Swedish created from recordings of Nordic parliamentary proceedings. It contains 900 hours of Finnish and 5 090 hours of Swedish speech, processed with an adapted Emilia pipeline and accompanied by unified evaluation sets for model development and benchmarking. The dataset aims to reduce the resource gap in TTS between high‑ and lower‑resourced languages.

By Zirui Li, Jens Edlund, Yicheng Gu, Nhan Phan, Lauri Juvela, Mikko Kurimo
arXiv Computation and Language
Oct 7

Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention

The study introduces MedQADE, a German open‑response clinical benchmark with 3,800 question‑answer pairs and physician reference annotations. It evaluates large language models (LLMs) as judges, finding that while some LLMs (e.g., Gemini 3 Flash) achieve physician‑level agreement on correctness, they exhibit self‑bias and low abstention rates. Physicians showed moderate agreement on correctness but limited agreement on difficulty, and their abstention increased with perceived difficulty.

By William Philipp, Finn Fassbender, Daniel Fister, Thorsten Langer, Martje G. Pauly, Rebecca Herzog, Markus A. Hobert, Theresa Paulus, Alexander Baumann, Chi Wang Ip, Lukas L. Goede, Johanna Reimer, Sebastian L\"ons, Ronald B\"ock, Sebastian Fudickar
arXiv Computer Vision
Oct 7

From the Drosophila Visual Connectome to General-Purpose Computer Vision

arXiv:2610.08418v1 Announce Type: new Abstract: Biological connectomes encode structured solutions to visual computation that may provide reusable inductive biases for artificial vision. We develop C...

By Zongyu Li, Akito Yamauchi, Huaizhi Liu, Vishwanatha Rao, Jia Guo, for the Frontotemporal Lobar Degeneration Neuroimaging Initiative, for the Alzheimer's Disease Neuroimaging Initiative