Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-ag...
arXiv:2606. 05384v1 Announce Type: new Abstract: LLM-as-judge evaluation is widely used in benchmarking pipelines, where model outputs are compared and ranked using automated evaluators.
arXiv:2606. 30887v1 Announce Type: cross Abstract: Large language models show promise for mental health support, yet therapeutic quality improves only when evaluation functions as an actionable control signal rather than a passive metric.
arXiv:2510. 10002v3 Announce Type: replace Abstract: As large language models (LLMs) are increasingly deployed in sensitive everyday contexts -- offering personal advice, mental health support, and moral guidance -- understanding their behavior in navigating complex moral reasoning is essential.
The paper evaluates the use of large language models (LLMs) as judges for assessing conversational voice agents, comparing human judgments with GPT‑4.1 and GPT‑5 across telecom and retail interactions. It examines agreement, metric‑level correlations, and consistency across three evaluation configurations (p0, p1, p2) to determine how reliably LLMs can judge conversational quality and safety. The study finds that LLM‑based evaluation can be effective but its reliability varies by metric and configuration, suggesting a hybrid approach where LLMs handle scalable assessment while humans focus on metrics requiring contextual interpretation.
arXiv:2608. 08210v1 Announce Type: new Abstract: Collaborative dialogue can end with apparent agreement while participants still differ on goals, assumptions, or execution plans, creating an \textbf{illusion of alignment (IoA)}.
arXiv:2605. 12530v2 Announce Type: replace-cross Abstract: LLM fairness should be evaluated through in-situ behavioral pattern rather than standardized-test Q&A benchmarks.
arXiv:2604. 02668v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often exhibit sycophancy: agreement with user stance even when it conflicts with the model's opinion.
The paper introduces multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Two initial methods—Score Averaging and Decision Voting—aggregate at the score and decision levels, respectively, and outperform single-expert approaches on homogeneous expert panels. To address limited coverage on heterogeneous panels, the authors propose Marginal‑Calibrated Conformal Consensus (MC3), which captures distinct per‑expert scoring scales through threshold ratios while maintaining a unified decision function, and demonstrate its effectiveness on a new 1,800‑pair human pairwise‑preference benchmark called Panel.
AgentJudgeBench is a new benchmark that evaluates the reliability of large language model (LLM) judges on agentic tool‑calling tasks involving workflow directed acyclic graphs (DAGs). It contains 3,808 instances across six DAG topologies and three difficulty tiers, tested with five generators (3B–70B open‑weight models and GPT‑5.4) and six judges (20B to frontier scale) under both paired‑with‑and‑without‑ground‑truth conditions. The study finds that judge alignment degrades with task difficulty, ground‑truth exposure can sometimes hurt alignment, and structured evaluation rubrics provide modest improvements, revealing a structural ceiling that model capacity alone cannot surpass.
arXiv:2601. 19921v2 Announce Type: replace-cross Abstract: Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simple majority vote despite higher computational cost.
arXiv:2606. 01552v1 Announce Type: new Abstract: Role-playing agents(RPAs) are widely used to steer large language models(LLMs) toward role-consistent behavior, yet existing benchmarks mainly evaluate surface-level fidelity and offer limited insight into decision making under role-alignment value conflicts.