Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning
Oct 22

Structured Temporal Causality for Interpretable Multivariate Time Series Anomaly Detection

Structured Temporal Causality for Interpretable Multivariate Time Series Anomaly Detection introduces OracleAD, an unsupervised framework that encodes each variable’s past into a causal embedding to jointly predict the present and reconstruct the input window. The embeddings are projected via self‑attention into a shared latent space, capturing dynamic spatial relationships that are aligned to a Stable Latent Structure (SLS) representing normal-state relationships. Anomalies are detected through a dual scoring mechanism based on prediction error and deviation from the SLS, allowing fine‑grained diagnosis and root‑cause identification at the variable level.

By Dongchan Cho, Jiho Han, Keumyeong Kang, Minsang Kim, Honggyu Ryu, Namsoon Jung
arXiv Computation and Language
Oct 22

Deep Reasoning in General Purpose Agents via Structured Meta-Cognition

arXiv:2605.11388v2 Announce Type: replace Abstract: Humans intuitively solve complex problems by flexibly shifting among reasoning modes: they plan, execute, revise intermediate goals, resolve ambigu...

By Dean Light, Michael Theologitis, Kshitish Ghate, Shuyue Stella Li, Benjamin Newman, Chirag Shah, Aylin Caliskan, Pang Wei Koh, Dan Suciu, Yulia Tsvetkov
arXiv Machine Learning
Oct 22

Estimating Performance for Safe Deployment of Machine Learning Models

The paper introduces MOVAL, a model evaluation tool that estimates performance of deep neural networks on unlabeled test data by calibrating class-wise confidence scores. MOVAL mitigates issues from distributional shifts and class imbalance, enabling estimation of metrics beyond accuracy such as F1-Score and AUC. Experiments on natural and medical image datasets demonstrate state‑of‑the‑art results across classification and segmentation tasks, providing accurate class, instance, and pixel‑level performance estimates for safe deployment.

By Zeju Li, Konstantinos Kamnitsas, Mobarakol Islam, Melanie Roschewitz, Cheng Ouyang, Chen Chen, Ben Glocker
arXiv Computation and Language
Oct 22

Evolving with Open Worlds: Towards Recursive Self-Improvement for Future Forecasting

arXiv:2610.29784v1 Announce Type: new Abstract: Most contemporary artificial intelligence (AI) systems are released with frozen parameters and improve through external updates, whereas an ideal syste...

By Zhipeng Chen, Yunsen Liang, Yanxi Chen, Hongkang Zhao, Lingyun Feng, Ruiyang Ren, Guoxin Chen, Jia Deng, Feng Chang, Chuan Hao, Ran Tao, Jian Yang, Ji-Rong Wen, Bryan Dai, Wayne Xin Zhao
arXiv Computer Vision
Oct 22

Rethinking Underwater Image Enhancement: Learning through Text-Guided Degradation Curriculum

The paper introduces a unified training framework for underwater image enhancement that addresses the scarcity of paired clean-degraded datasets. It employs a text-guided diffusion model to generate diverse, semantically consistent degraded images and uses a progressive degradation curriculum to balance learning across multiple attributes and difficulty levels. The approach also incorporates a transmission-guided one-step diffusion model to handle varying degradation severity, achieving superior performance on several underwater benchmarks.

By Hongri Tian, Wenxue Li, Tian Ye, Sixiang Chen, Lujia Wang, Xianying Wang, Xinhu Zheng, Gang Hua, Lei Zhu
arXiv Computer Vision
Oct 22

Open-Vocabulary Models and Label Propagation for Uncertainty-aware Pseudo-Labeling of Object Detection Datasets

The paper presents a semi‑supervised framework that uses open‑vocabulary object detectors to generate region proposals and semantic representations, builds an object‑level similarity graph, and propagates probabilistic labels to assign soft, uncertainty‑aware annotations to previously unlabeled objects. Experiments on four detection benchmarks show that high‑quality soft labels can be produced with only a small fraction of manual annotations, and models trained on these semi‑automatically annotated datasets achieve performance comparable to fully manually annotated models. On COCO, the method reaches 40.5% AP, nearly matching the 40.9% AP of a fully annotated model while incurring only 1% of the labeling cost.

By Jonathan Klees, Timo Heller, Matthias Rottmann