arXiv:2507.21831v2 Announce Type: replace-cross Abstract: LLMs are seeing widespread use for task automation, including automated coding in the social sciences. However, even though researchers have...
arXiv:2605. 11954v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used in social science as scalable measurement tools for converting unstructured text into variables that can enter standard empirical designs.
arXiv:2609.23742v1 Announce Type: new Abstract: Small open-source large language models (LLMs) in the 0.6B-4B parameter range are increasingly deployed for structured output generation (JSON, functio...
Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart.
arXiv:2609.06993v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly generate Markdown that is consumed by renderers, agents, code extractors, and structured downstream pipelin...
The paper introduces DECO, a diagnostic tool that factorises content into independent criteria for evaluating large language models (LLMs) on content moderation tasks. Using DECO and pairwise evaluation across four datasets and four LLMs, the authors find that high benchmark scores can mask significant failures at the criterion level, especially when decisions hinge on specific content aspects rather than overall harmfulness. The study underscores that aggregated label performance does not guarantee reliable criterion-conditioned evaluation, calling for new methods that explicitly assess this behavior.
arXiv:2608. 16643v1 Announce Type: cross Abstract: Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation.
arXiv:2608.22138v1 Announce Type: new Abstract: Language models are usually judged by a single accuracy score, which does not reveal how their performance degrades as inputs are perturbed. We present...
The paper introduces DECO, a diagnostic framework that factorises content into independent moderation criteria, allowing controlled evaluation of large language models (LLMs) at the criterion level. Using pairwise evaluation across four datasets and four LLMs, the authors find that high aggregate benchmark scores can mask significant failures when decisions hinge on specific content aspects required by individual criteria. The study underscores that aggregated labels do not guarantee reliable criterion-conditioned performance, highlighting the need for evaluation methods that explicitly assess this behavior.
arXiv:2504.18346v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have been transformative across many domains. However, hallucination, i.e., confidently outputting incorrect inf...
arXiv:2609.24516v1 Announce Type: new Abstract: In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these s...
Large Language Models (LLMs) are being tested as a cost‑effective alternative to human judgment for evaluating machine translation (MT). This study compares LLM agreement with human annotators on two MT quality schemes—Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA)—across 70 language pairs and public WMT23/25 data. Results show that while LLMs sometimes match or exceed human agreement, performance varies widely by scheme, language, and domain, and remains unreliable in many settings. The work also identifies specific challenges for both humans (e.g., fine‑grained MQM, low‑resource languages) and LLMs (e.g., minor errors, wrong language variants, span annotation).