arXiv Machine Learning
Verdict Instability of OOD Scores under Reference Resampling
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 25
Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence
The paper introduces the Wiggle Framework, a unified stress test for assessing epistemic stability in large language model (LLM) judges. It evaluates judge robustness across three dimensions—Mechanical Consistency, Single-turn Conviction, and Multi-turn Persistence—using 9 frontier models on 14 judging tasks related to safety, toxicity, AI writing detection, and political-response evaluation. Results show significant instability, with verdict flips ranging from 25–71% under static pushback and 62–91% when challenged by an adversarial LLM, and highlight that successful pressure often misaligns with ground truth.
arXiv AI
Aug 14
Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence
arXiv:2608. 12645v1 Announce Type: new Abstract: LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling.
arXiv AI
Jun 16
Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
arXiv:2606. 15474v1 Announce Type: new Abstract: Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down.
arXiv Machine Learning
Jun 9
LOTTERY: Learning from Reference-Only Samples in Two-Sample Testing under Size Asymmetry
arXiv:2606. 08460v1 Announce Type: cross Abstract: Data-adaptive two-sample testing assesses if two samples come from the same distribution, using a discrepancy learned from the data (e.
arXiv AI
1d ago
How Much Can Reliability Drift Under a Fixed Confidence Distribution?
arXiv:2609.38917v1 Announce Type: new Abstract: A classifier's conditional accuracy can change while its confidence distribution stays exactly the same. We study the worst-case movement of the reliab...
arXiv AI
Jul 1
RoPoLL: Robust Panel of LLM Judges
arXiv:2606. 30931v1 Announce Type: new Abstract: The LLM Jury, a Panel of LLM Evaluators (PoLL) reporting consensus scores, has become a practical alternative to single-judge LLM evaluation, yet its statistical behavior remains poorly understood.