JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols

JudgeStealer is a query‑efficient framework that extracts the judging capabilities of large language models across pointwise scoring, pairwise comparison, and listwise ranking protocols. It leverages cross‑protocol agreement to convert pointwise scores into higher‑order supervision, dynamically selects informative inputs, and applies score smoothing and multi‑protocol review to preserve ordinal structure and avoid catastrophic forgetting. Experiments show it outperforms existing baselines, achieving up to 73.3% accuracy on pointwise, 87.0% on pairwise, and 71.6% on listwise evaluation, while remaining robust against common extraction defenses.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• benchmarks