← Back to all news
arXiv AI By Vaibhava Lakshmi Ravideshik, Mayank Kejriwal

Reliability of LLM Judges for Evaluating Entity Alignment

Read the original on arXiv AI →

The paper investigates how reliable large language models (LLMs) are when used as judges to evaluate entity alignment (EA) systems, which match equivalent entities across knowledge graphs. Through systematic benchmarking across three frontier LLMs, three datasets, and four EA systems, the study identifies a failure mode called anchor bias, where judges invert discrimination when the system’s decision label is visible. The authors show that a label‑free evaluation protocol restores high discrimination accuracy, while visible labels cause significant score shifts, especially in stronger models—a phenomenon they term the frontier model paradox. They also provide a new biomedical EA benchmark (MeSH–SNOMED CT) and an open‑source framework for auditing LLM judge reliability.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 14

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

The paper investigates how large language models (LLMs) used as judges in absolute scoring tasks exhibit systematic biases that compromise reliability. It shows that a judge’s task accuracy strongly predicts both its judging accuracy and its directional bias, yet more capable examinee models consistently receive more lenient judgments. To mitigate these biases, the authors propose a calibrated weighted majority voting (WMV) ensemble that estimates judges’ error rates from inter-judge agreement patterns, achieving near-oracle performance without labeled data and improving both accuracy and fairness.

By Gemma Zhang, Prachi Badarayani, Asmi Kumar, Sadid Hasan, Sulaiman Vesal
llmsbenchmarkssafety
More like this →
arXiv Computation and Language
Sep 2

Post-hoc Alignment of LLM-judges to Human Judgment Distribution

The paper introduces NAPHA, a lightweight post‑hoc alignment method that improves large language model (LLM) predictions of human judgment distributions (HJD) by matching LLM output distributions to HJD through entropy‑based class assignment and specialized alignment models. Experiments on five datasets show that while LLMs perform near human‑level on hard‑label tasks, they struggle with soft‑label predictions, and NAPHA consistently enhances soft‑label accuracy, especially on high‑entropy instances. The study also demonstrates that better entropy class prediction can further boost NAPHA’s effectiveness.

By Sebastian Steindl, Nikos Voskarides, Alberto Gasparin, Diego Marcheggiani
llmssafety
More like this →