arXiv:2608.29517v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as essay graders in learning analytics, evaluated almost exclusively with agreement statistics. Educ...
arXiv:2606. 15887v1 Announce Type: cross Abstract: Large language model (LLM) systems are increasingly proposed to assist peer review, yet most evaluations judge the prose of machine-generated review text, not the validity of the numeric score a system assigns.
arXiv:2609.23264v1 Announce Type: new Abstract: Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high sc...
arXiv:2608. 03659v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting.
arXiv:2609.14963v1 Announce Type: new Abstract: As large language models become capable translators of classical texts, a key challenge is deciding which outputs need expert review when no human refe...
arXiv:2609.13824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faste...
The paper "What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks" analyzes 14,767 arXiv submissions from 2022 to 2026 that introduce or update evaluation resources for large language models. It systematically maps changes in target systems, domains, evaluation materials, conditions, and scoring mechanisms, revealing a growing emphasis on action, interaction, and professional applications. The study also notes uneven development in model participation, with LLM-based scoring increasing in both agent and non-agent groups, while model-generated materials do not show a comparable rise.
The paper examines whether removing declared language fields from de‑identified résumés eliminates demographic leakage in large language models. By keeping language attributes identical and varying only unstructured prose across five ethnocultural groups and three cue‑salience levels, the authors find that non‑language text still allows target‑group recovery (average 0.757, reaching 1.000 under high salience). They also show that evaluation design—such as allowing or forbidding ties—dramatically affects LLM‑as‑a‑judge outcomes, underscoring the importance of evaluation protocol in bias audits.
arXiv:2610.20417v1 Announce Type: new Abstract: Audits of large language model (LLM) extraction from full-length documents report severe unreliability, leaving two questions open: what the failures a...
The study examines whether asking large language models (LLMs) to predict whether readers would share misinformation adds useful information beyond predicting credibility. Eight LLM versions were audited on 290 synthetic misinformation articles, yielding 1,256 paired survey responses. While credibility scores explained a substantial portion of sharing variation, adding sharing predictions to credibility did not meaningfully reduce error, indicating that sharing answers contain structured variation but lack established incremental validity for human sharing.
The study investigates whether a second review by a different large language model (LLM) improves error detection in code, documentation, and analyses. Using 30 artifacts with 150 planted errors and 900 review sessions across three reviewer models, the authors find that a top-tier cross-model reviewer performs similarly to same-model reviews in fresh sessions, identifies partly different errors, and that a combination of a fresh same-model review and a cross-model review detects more errors than two fresh same-model reviews. However, the results do not conclusively separate the effects of model difference from reviewer capability, and a lightweight cross-model reviewer does not outperform same-model review. whyItMatters":"The findings suggest that while cross-model reviews can enhance error detection, the benefit depends on reviewer skill and the specific review strategy, informing how teams might structure LLM-based code review processes."
arXiv:2608. 08975v1 Announce Type: cross Abstract: As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions.