arXiv AI By Cheng Chang, Yining Mao, Peng Qi

PADM\'E: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

Read the original on arXiv AI →

PADM'E is a method for synthesizing preference‑aligned data to meta‑evaluate language‑model (LM) evaluators of agentic behaviors. It reframes meta‑evaluation as a preference judgment problem, generating criterion‑based data with small LMs and no human involvement. In a prototype, PADM'E produced 1,000 samples across four domains and three criteria, and human validation showed agreement with human judgment rising from 73% to 85% compared to a naive baseline.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 31

Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing

The paper introduces the "Prompt Lottery" phenomenon, where paraphrastic rephrasings of judge prompts in LLM-as-a-Judge (LLMaJ) lead to inconsistent judgments and can even reverse model rankings. A large-scale study across various tasks, judge families, and model backbones demonstrates that these prompt variations significantly alter evaluation outcomes. A user study shows that LLMaJ judges are 6–9 times more sensitive to prompt rephrasings than humans, and that aggregating multiple prompts improves robustness by focusing on consistent judgments.

By Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen
Hugging Face Trending Papers
Oct 13

Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

The paper reproduces five factuality alignment methods for large language models—three on‑policy RL approaches (PPO, GRPO, Dr.GRPO) and two offline preference optimization methods (DPO, KTO)—using a common benchmark and evaluation metrics. None of the methods consistently outperforms the base model, and the rewards appear to favor format, length, or brevity rather than factuality or relevance. The study also highlights problems with current evaluation practices, noting that metrics and benchmarks often fail to measure prompt relevance and rely on fragile LLM judges and retrieval for verification.

arXiv Computation and Language
Aug 27

Lower-Resource, Higher Scores: Language Bias in LLM Evaluators

The paper demonstrates that large language model (LLM) evaluators, whether reward‑model based or prompted LLM‑as‑a‑Judge, exhibit significant language bias in multilingual settings. Experiments with semantically identical instruction‑response pairs across 23 languages reveal that lower‑resource languages receive higher scores, a bias that persists across eight open‑weight evaluators and is not detectable by standard pairwise accuracy metrics. The authors link the bias to model uncertainty and language identity, showing it cannot be explained by content difficulty alone.

By Ej Zhou, Lucas Resck, Zheng Hui, Anna Korhonen