arXiv Machine Learning
Auditing Bayesian Graph Alignment: Diagnostic Comparisons and Reference Failure
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 31
Reliability of LLM Judges for Evaluating Entity Alignment
The paper investigates how reliable large language models (LLMs) are when used as judges to evaluate entity alignment (EA) systems, which match equivalent entities across knowledge graphs. Through systematic benchmarking across three frontier LLMs, three datasets, and four EA systems, the study identifies a failure mode called anchor bias, where judges invert discrimination when the system’s decision label is visible. The authors show that a label‑free evaluation protocol restores high discrimination accuracy, while visible labels cause significant score shifts, especially in stronger models—a phenomenon they term the frontier model paradox. They also provide a new biomedical EA benchmark (MeSH–SNOMED CT) and an open‑source framework for auditing LLM judge reliability.
Hugging Face Trending Papers
Oct 7
Reliability of LLM Judges for Evaluating Entity Alignment
Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expe...
arXiv AI
Sep 25
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
The paper investigates the reliability of ranking tables produced by small-sample evaluations of large language models (LLMs). Using LLM‑inferred prompt structure across eight model variants, the authors find that prompt‑structure recovery is highly unstable, with only the bottom of the ranking consistently reproducible. They demonstrate that standard evaluation practices can misrepresent model performance and propose reporting practices to improve transparency.
arXiv Computation and Language
Oct 12
AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.
arXiv AI
Jul 29
GraphRareBench: An Auditable Graph-Evidence Benchmark for Phenotype-Driven Rare-Disease Diagnosis
arXiv:2607. 24878v1 Announce Type: cross Abstract: Phenotype-driven diagnostic benchmarks usually report the rank of the reference disease, but they rarely reveal which plausible alternatives are ranked above it or what evidence a tool-using model examines before making its decision.
arXiv AI
Jul 10
When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.