Hugging Face Trending Papers
Stopping and Routing LLM Judge Panels
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Computation and Language
Aug 21
Stopping and Routing LLM Judge Panels
arXiv:2608. 19802v1 Announce Type: new Abstract: LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers.
arXiv AI
Oct 2
False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift
The paper examines safety routers—systems that route user requests to different language models—and finds that their performance degrades significantly when evaluated under distribution shift. In standard benchmarks, routers appear effective because the best single model is chosen from the same evaluation data, but when the data distribution changes, the routing advantage diminishes or disappears. The study quantifies this bias across multiple safety corpora, showing that routers offer little benefit under realistic shift conditions and that recognition‑based defenses can be undermined by attackers who know the model being used.
arXiv AI
Sep 30
LLM Judge Validation Under Sparse Overlap: From Inference to Design
arXiv:2609.31857v2 Announce Type: replace Abstract: Validating an LLM-as-a-judge requires estimating its agreement with humans, yet annotation budgets rarely allow every item to be multiply labeled....
arXiv AI
Aug 24
Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation
The paper investigates two strategies for improving large language model (LLM) evaluation: specialized judge weights and rule‑based deferral policies. Experiments on nearly 100,000 rubric‑conditioned samples show that correct rubrics boost accuracy, while incorrect ones hurt it, and that splitting training data into criterion‑specific experts can severely degrade performance unless the experts are warm‑started from a unified model. The authors demonstrate that lightweight deferral cascades can match or exceed the accuracy of larger standalone judges at a fraction of the compute cost, and they provide practical design rules for building efficient, reliable LLM evaluators.
Hugging Face Trending Papers
Jul 30
Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation
Agentic systems have widened the gap between producing candidate outputs and reviewing them. This paper asks a practical architectural question: should domain specialization be built into an evaluator's weights, or into the rule that decides when its judgment can be trusted?
arXiv AI
Jul 9
Operational Reframing and Approval-Framed Delegation in Multi-Agent LLM Safety
arXiv:2607. 07097v1 Announce Type: new Abstract: Safety evaluations of multi-agent LLM systems often compare a direct prompt with a planner-executor pipeline and report the difference as a single "pipeline effect.