The paper introduces a Chinese competitive debating dataset and benchmark, comprising 148 fully adjudicated matches with 2,698 stages and 20,542 exchange units, all annotated by 120 professional judges using a shared rubric. It offers manually verified transcripts, segmentation, and preserved scores, votes, and rationales, enabling evaluation of large language models on winner-tendency prediction, stage-score prediction, and best-debater prediction. Zero‑shot experiments show the best models achieve 66.2% winner‑prediction accuracy, a 0.250 Pearson correlation for stage scores, and 56.8% best‑debater accuracy.
arXiv:2305.12474v4 Announce Type: replace Abstract: Large Language Models(LLMs) have demonstrated remarkable performance across various natural language processing tasks; however, how to comprehensiv...
The paper introduces UPHELD, a large benchmark of human-to-human dialogues written by professional script writers, featuring realistic turn densities and over 36,000 per-turn human annotations. It evaluates existing automatic metrics and LLM-as-a-judge methods, finding them unreliable against expert human judgment. Using UPHELD, the authors develop a Mixture-of-Judges framework that improves correlation with human assessments by about 30%.
arXiv:2605.28313v2 Announce Type: replace Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in tasks related to reasoning and judgment. However, assessing the quality o...
arXiv:2605. 15416v2 Announce Type: replace-cross Abstract: Jung et al.
The paper introduces JudgeBiasBench, a benchmark that systematically quantifies judgment biases in large language model (LLM)-based judges across four dimensions and 12 bias types. It evaluates both generative and discriminative judges, revealing significant bias patterns that undermine reliability. The authors propose bias-aware training—reinforcement learning for generative judges and contrastive learning for discriminative judges—to reduce these biases while maintaining evaluation performance.
arXiv:2601. 16217v2 Announce Type: replace-cross Abstract: Large language models increasingly mediate multilingual professional communication, where useful generation requires adapting to community conventions about which expressions are retained, translated, or mixed.
arXiv:2508. 03250v4 Announce Type: replace-cross Abstract: The increasing amount of political debates and politics-related discussions calls for the definition of novel computational methods to automatically analyse such content with the final goal of lightening up political deliberation to citizens.
arXiv:2606. 18699v1 Announce Type: cross Abstract: Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored.
Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating large language models remain limited. We present T...
Large Language Models (LLMs) as judges across various scenarios such as assessing model responses is becoming an increasingly accepted paradigm. However, existing judgment approaches often rely on trained judgers using fixed preference data, which tend to overlook diverse user preferences and struggle to adapt to real-world human-AI dialogue scenarios.
The paper introduces STAR‑Ar, a BERT‑BiLSTM‑CRF model designed for the Daleel 2026 Arabic argument mining shared task. It treats argument discourse unit detection and classification as a token‑level sequence labeling problem, achieving an F1‑score of 72.69 on validation and 73.7 on test data. Analysis shows that models trained only on editorial texts perform worse than those trained on debates, mainly due to the smaller editorial dataset.
arXiv:2609.18156v1 Announce Type: new Abstract: Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating l...