arXiv AI
5d ago
Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence
arXiv:2608. 12645v1 Announce Type: new Abstract: LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling.
Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges
arXiv:2606. 05384v1 Announce Type: new Abstract: LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators.
Related stories
arXiv AI
Jul 1
Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support
arXiv:2606. 30887v1 Announce Type: cross Abstract: Large language models show promise for mental health support, yet therapeutic quality improves only when evaluation functions as an actionable control signal rather than a passive metric.
arXiv AI
Jun 15
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
arXiv:2606. 13685v1 Announce Type: cross Abstract: LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized.
arXiv AI
Jun 3
Quantifying and Mitigating Self-Preference Bias of LLM Judges
arXiv:2604. 22891v4 Announce Type: replace-cross Abstract: LLM-as-a-Judge has become a dominant approach in automated evaluation systems, playing critical roles in model alignment, leaderboard construction, quality control, and so on.
arXiv AI
Jul 10
When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
arXiv AI
Jul 31
Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control
arXiv:2607. 01153v3 Announce Type: replace-cross Abstract: Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model followed an instruction, refused appropriately, complied with a policy, or misreported progress in an agentic task.
Hugging Face Trending Papers
1d ago
Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
Using LLMs as judges has become standard practice for evaluating model outputs at scale. This is particularly common for subjective, open-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists.
arXiv AI
Jun 24
AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability
arXiv:2606. 24589v1 Announce Type: new Abstract: Scaling adversarial evaluation of large language models requires both a method for generating hard inputs and a reliable way to confirm that resulting failures are real.
arXiv AI
Jun 2
STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems
arXiv:2605. 02122v2 Announce Type: replace-cross Abstract: Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation.
arXiv AI
Jul 29
Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
arXiv:2607. 25136v1 Announce Type: new Abstract: Research on preference optimization often varies the training objective while holding the data fixed.
arXiv AI
Aug 10
Divergent Response Modes in Frontier Language Models Under Steering Pressure
arXiv:2608. 06578v1 Announce Type: new Abstract: Frontier language models are trained using distinct data, objectives, and safety pipelines.
Hugging Face Trending Papers
Jul 8
Online Data Selection Is Implicit Alignment
Supervised fine-tuning (SFT) is often treated as a capability-adaptation step, while alignment is attributed to later preference optimization or reinforcement learning. This separation is incomplete: when examples are scored and kept online during fine-tuning, the choice of which data to train on already changes the model's behavioral preferences.