arXiv AI
Jun 10
Flaws in the LLM Automation Narrative
arXiv:2606. 11166v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly described as performing at the level of human experts on knowledge economy tasks.
What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks
The paper "What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks" analyzes 14,767 arXiv submissions from 2022 to 2026 that introduce or update evaluation resources for large language models. It systematically maps changes in target systems, domains, evaluation materials, conditions, and scoring mechanisms, revealing a growing emphasis on action, interaction, and professional applications. The study also notes uneven development in model participation, with LLM-based scoring increasing in both agent and non-agent groups, while model-generated materials do not show a comparable rise.
Related stories
arXiv Computation and Language
5d ago
LLJ Cards: Best practices for the Use of LLMs as Judges
arXiv:2609.24516v1 Announce Type: new Abstract: In recent years, large language models (LLMs) have emerged as a popular alternative for evaluation. Often referred to as LLMs as judges (LLJs), these s...
arXiv AI
4d ago
Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluation
arXiv:2609.23201v1 Announce Type: new Abstract: Benchmark scores increasingly influence the development, marketing, and selection of large language models (LLMs). Yet an overall score is interpretabl...
arXiv AI
Sep 3
EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models
EvalDetectBench is an open pipeline and benchmark designed to measure evaluation awareness in frontier large language models, enabling practitioners to test models against any Inspect-compatible evaluation. It includes a curated transcript suite from current frontier system-card evaluations and diverse deployment sources, and it assesses both how reliably models recognize they are being evaluated and how detectable individual benchmarks are. The benchmark addresses systematic bias by calibrating probes per model and harmonizing generator selection to correct for variance caused by model identity and prompt choice.
arXiv Machine Learning
Aug 4
Computational Approaches to Understanding Large Language Model Impact on Writing and Information Ecosystems
arXiv:2506. 17467v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown significant potential to change how we write, communicate, and create, leading to rapid adoption across society.
arXiv AI
Aug 25
What Proves You Wrong: Benchmarking Language Models on Falsifiable Research Ideation
The paper introduces Lit2Test, a benchmark that evaluates language models’ research idea proposals by requiring each idea to include a falsifiable outcome, thereby making quality decidable. Built from 200 real-paper neighborhoods, the benchmark gathers proposals from four frontier models and compares them via 1,200 blind pairwise judgments, with reliability checks and human calibration. The results show a consistent ranking of the models, driven by test and metric quality rather than fluency, and the authors release the benchmark and related artifacts for public use.
arXiv AI
Jul 7
Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility
arXiv:2501. 10711v5 Announce Type: replace-cross Abstract: Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities.
arXiv Computation and Language
Aug 27
Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
The study investigates how prior scores influence large language model (LLM) judgments in the LLM-as-a-Judge paradigm. By testing three prompt conditions—no metadata, revision framing, and anchored metadata containing prior scores—the authors find that prior scores systematically bias evaluations, shifting ratings toward those scores across 192,000 attempts. The bias also affects categorical decisions, blocking 48% of error corrections and flipping 10.18% of correct judgments, and is not mitigated by Chain-of-Thought or a warning, underscoring the need for careful context engineering.
arXiv AI
Aug 3
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
arXiv:2607. 28801v1 Announce Type: cross Abstract: Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples.
arXiv Machine Learning
Jun 8
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.
arXiv AI
Sep 1
Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
Ideation Arena is a battle-style platform that evaluates research ideas generated by large language models (LLMs) and research agents through pairwise human assessment. The system builds shared literature contexts, collects over 6,000 double-blind comparisons from 105 computer science researchers, and constructs an Elo rating leaderboard to rank proposal-stage expert preferences. It also introduces Ideation Arena Eval, a benchmark to test whether automated evaluators align with human preferences, finding that current LLM judges achieve at best 72.56% Soft Accuracy on overall quality.
arXiv AI
Jun 9
Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
arXiv:2606. 08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes.