arXiv AI
Aug 7
Stability of Ranking-dependent Pair-wise Comparison Patterns in the Analytic Hierarchy Process
arXiv:2608. 05958v1 Announce Type: new Abstract: The paper addresses several ranking-dependent decision support methods.
Integrating the Analytic Hierarchy Process with Large Language Models for Transparent Multi-Criteria Decision-Making
Related stories
arXiv AI
Aug 6
EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks
arXiv:2608. 04549v1 Announce Type: cross Abstract: Frontier LLMs are increasingly put to use on open-ended complex questions, different in nature from the ones they are typically evaluated on.
arXiv AI
Aug 11
WuYuEval: A Multi-Level Benchmark for Large Language Models in Solid Waste Management
arXiv:2608. 07529v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as technical assistants, but their competence in solid waste management (SWM) remains difficult to assess because existing benchmarks emphasize general knowledge rather than professional decisions under engineering, environmental, and policy constraints.
arXiv AI
Jun 3
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
arXiv:2605. 25240v2 Announce Type: replace-cross Abstract: Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs.
arXiv AI
Aug 21
Qworld: Question-Specific Evaluation Criteria for LLMs
arXiv:2603. 23522v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question's context.
arXiv AI
Jun 2
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
arXiv:2605. 28183v2 Announce Type: replace-cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law.
arXiv AI
Jul 1
Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
arXiv:2605. 24661v2 Announce Type: replace Abstract: LLMs have achieved remarkable success in complex reasoning tasks, yet current evaluation approaches predominantly rely on final-answer correctness, offering limited insight into the underlying reasoning processes that produce those answers.
arXiv Computation and Language
Sep 10
From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
arXiv:2601.08654v3 Announce Type: replace Abstract: Rubric-based text evaluation increasingly relies on large language models (LLMs) as scalable judges, yet frozen black-box models can interpret the...
arXiv AI
Jun 18
TW-LegalBench: Measuring Taiwanese Legal Understanding
arXiv:2606. 18699v1 Announce Type: cross Abstract: Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored.
arXiv Machine Learning
Jul 31
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
arXiv:2607. 28282v1 Announce Type: cross Abstract: Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive.
arXiv Computation and Language
Sep 1
LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks
arXiv:2606.09389v2 Announce Type: replace Abstract: As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses...
arXiv AI
Aug 20
SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
The paper introduces SESSE, a training‑free framework that breaks down LLM‑as‑judge evaluations into five steps—Sketch, Expand, Sort, Summarize, Evaluate—by mining sub‑questions from the judge’s own error cases. It requires no oracle responses, task‑specific rubrics, or fine‑tuning, yet on RewardBench it matches the performance of chain‑of‑thought baselines and rivals a fine‑tuned specialist (RISE‑Judge‑32B). SESSE provides per‑criterion vote evidence, offering an interpretable audit trail that can diagnose label ambiguity and judge failure modes that a single holistic output token cannot reveal.