arXiv AI
PRISM: Calibrating Black-Box Judges with Counterfactual Probes
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 10
From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
arXiv:2601.08654v3 Announce Type: replace Abstract: Rubric-based text evaluation increasingly relies on large language models (LLMs) as scalable judges, yet frozen black-box models can interpret the...
arXiv Computation and Language
Sep 16
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
The paper introduces JudgeBiasBench, a benchmark that systematically quantifies judgment biases in large language model (LLM)-based judges across four dimensions and 12 bias types. It evaluates both generative and discriminative judges, revealing significant bias patterns that undermine reliability. The authors propose bias-aware training—reinforcement learning for generative judges and contrastive learning for discriminative judges—to reduce these biases while maintaining evaluation performance.
Hugging Face Trending Papers
Oct 2
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
Large language models (LLMs) are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs...
arXiv Machine Learning
Oct 23
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
The study investigates how different prompting strategies affect large language model (LLM) judges across multiple languages and tasks. Using a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks, the authors collected over 1 million pointwise scores validated against human judgments. Results show that prompting effectiveness is highly dependent on the judge model, can reverse direction across judges, and becomes less stable in low‑resource languages, indicating that prompt choice must be tailored to the specific evaluation setting.
arXiv Machine Learning
Jun 5
A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.
arXiv AI
2d ago
Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing
The paper introduces the "Prompt Lottery" phenomenon, where paraphrastic rephrasings of judge prompts in LLM-as-a-Judge (LLMaJ) lead to inconsistent judgments and can even reverse model rankings. A large-scale study across various tasks, judge families, and model backbones demonstrates that these prompt variations significantly alter evaluation outcomes. A user study shows that LLMaJ judges are 6–9 times more sensitive to prompt rephrasings than humans, and that aggregating multiple prompts improves robustness by focusing on consistent judgments.