← Back to all news
arXiv AI By Lining Hu, Ting Liu, Yuzhuo Fu

What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarks

Read the original on arXiv AI →

The paper examines how pooled leaderboards in microservice root cause analysis (RCA) benchmarks obscure key details about which subsystems the rankings apply to and how scores are generated. By auditing three public benchmark families (OpenRCA, RCAEval, and PetShop) and running 16 released methods on 11 subsystems, the authors show that many reported leads do not hold across all subsystems and that differences in input tables and implementation details can reverse method rankings. They find that none of the 23 RCA papers examined provide the per‑case outputs needed to verify these claims, and they recommend that future pooled comparisons disclose subsystem coverage, score spread, input tables, handling of failures, and denominators, along with per‑case outputs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 24

Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

The paper investigates whether graph structure improves microservice root‑cause analysis by conducting a controlled study on the RCAEval benchmark. Using identical features, optimizers, and evaluation protocols across three model variants, the authors find no consistent advantage for graph‑based models over flat models, with a negligible Avg@5 difference (0.003, p=0.844). They identify two benchmark properties—limited fault injection and a non‑uniform telemetry schema—that bias results, and propose a new model, PSC‑GRCA, which achieves higher Avg@5 mainly through a system prior rather than graph information.

By Imad Bulji\'c
benchmarks
More like this →
arXiv Machine Learning
6d ago

StrictPRMBench: A Replayable Evaluation Benchmark for Auditing Fixed-Score PRM Aggregation

StrictPRMBench is a replayable evaluation benchmark designed to audit fixed‑score PRM aggregation rules. It standardizes mathematical reasoning pools, scorer evidence, and deletion interventions so that different reducers confront identical selection problems. Across seven checkpoints and two benchmarks, the benchmark reveals that product aggregation yields higher deletion‑induced selection instability than geometric mean, with statistically significant differences in most cells.

By Pengfei Zheng
benchmarks
More like this →