arXiv AI
Jul 23
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
arXiv:2604. 06996v2 Announce Type: replace-cross Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs.
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
arXiv:2605. 25240v2 Announce Type: replace-cross Abstract: Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs.
Related stories
arXiv AI
Jun 2
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
arXiv:2605. 28183v2 Announce Type: replace-cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law.
arXiv AI
Jul 3
AI Assistance for Human Review of Default Judgments
arXiv:2607. 01256v1 Announce Type: cross Abstract: Overwhelmed courts in the United States review millions of default judgments each year.
arXiv AI
Jun 18
TW-LegalBench: Measuring Taiwanese Legal Understanding
arXiv:2606. 18699v1 Announce Type: cross Abstract: Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored.
arXiv AI
Jun 8
HKJudge: A Legal Discourse-Annotated Corpus for Interpreting What Courts Find, How They Reason, and What They Rule
arXiv:2606. 06679v1 Announce Type: cross Abstract: Court judgments are central to legal practice and jurisprudence, yet discourse analysis of Hong Kong judgments has received limited attention, owing largely to the absence of expert-annotated corpora.
arXiv AI
Jul 7
From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Control
arXiv:2607. 03325v1 Announce Type: cross Abstract: We present an automated pipeline that decomposes Italian tax-court judgments into individual legal issues and extracts, for each issue, a structured XML representation grounded in the IRAC framework and the legal syllogism.
arXiv AI
Jun 15
Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
arXiv:2604. 14892v3 Announce Type: replace-cross Abstract: Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators.
arXiv AI
Jul 10
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
arXiv:2607. 08535v1 Announce Type: cross Abstract: An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed.
arXiv Machine Learning
Jul 3
Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges
arXiv:2607. 02104v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers.
arXiv AI
Jun 3
Quantifying and Mitigating Self-Preference Bias of LLM Judges
arXiv:2604. 22891v4 Announce Type: replace-cross Abstract: LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on.
arXiv AI
Jun 2
On Wednesdays, We Ask Questions: Optimizing "Active Listening" in Automated Legal Triage and Referral
arXiv:2606. 00272v1 Announce Type: new Abstract: The FETCH classifier generates follow-up questions to help refine the best match for the applicant's legal problem, using a low-cost ensemble of LLMs.
arXiv AI
6d ago
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
arXiv:2608. 12097v1 Announce Type: new Abstract: Rubric-based evaluators commonly treat rubrics as prompt context or flat criteria: they specify what to judge but leave criterion composition implicit, even when natural-language rules state it.