The paper addresses the disconnect between textual data quality (DQ) dimensions and practical DQ tools by making ISO/IEC 25024 and ISO/IEC 5259 metrics executable. It classifies metrics into generalizable, parameterized, and non-generalizable categories, and provides implementation guidelines for the first two. The authors implement these guidelines in the open-source library dqmeasure, demonstrating that the metrics respond predictably to injected errors, correlate with downstream machine learning performance, and scale linearly with dataset size.
arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.
The paper introduces LeDQeR, an approach that uses large language models to automatically generate data quality rules for rule‑based enterprise tools. It follows a generate‑filter framework where the LLM proposes candidate rules from dirty data, and four filters ensure the rules are executable, correct, generalizable, and non‑redundant. Experiments show that LeDQeR produces effective, compact rule sets across diverse datasets and error types.
arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.
Litmus is a zero-label system that automatically designs evaluation and monitoring metrics for AI pipelines by extracting evaluation intent from source code and targeted interrogation. It identifies what must be measured and why, then generates a justified, per-stage metric portfolio. In experiments on three real code-defined pipelines, Litmus outperforms AutoMetrics and DynamicRubric baselines in concern coverage, stage span, redundancy, and validity against per-row quality labels, especially on scientific QA.
The paper introduces AXIOM, a perturbation-based framework for creating a balanced benchmark of code evaluation problems. AXIOM refines program quality by applying controlled semantic perturbations and then calibrates scores through selective manual annotation, resulting in the multilingual AXIOMBENCH dataset with 1,962 problems. This approach addresses limitations of prior benchmarks that rely on coarse binary labels or subjective human scoring.
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
arXiv:2610.17358v1 Announce Type: cross Abstract: Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author mean...
The paper argues that calibration—how well a language model’s confidence aligns with its actual correctness—should be a standard evaluation metric for large language models (LLMs). It notes that while calibration metrics exist, they are rarely applied outside specialized NLP subfields, leading to unverified confidence scores in new models, datasets, and benchmarks. The authors highlight the risks of miscalibration both at deployment (overconfident errors causing harm) and in research workflows (affecting LLM-as-a-judge, synthetic data generation, and active learning). They call for every NLP subfield to pair its primary performance metric with a calibration score, treating calibration as an essential property of every model.
arXiv:2607. 22766v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality.
The paper introduces an explainable, header‑centric framework for semantic table interpretation and data quality assessment that operates solely on column headers. It maps headers to 39 interpretable types using curated lexical resources, applies validation rules to detect data quality issues, and aggregates these into a lightweight quality metric called HeadersIQ. The framework was evaluated on diverse benchmarks, demonstrating broad coverage of noisy real‑world metadata and providing a workflow for metadata‑driven semantic annotation, quality monitoring, and benchmark diagnosis.
The paper presents a unified dataset of 350 Lean tasks to evaluate specification generation (SpecGen) for verifiable code, combining metrics of formal validity, reference similarity, equivalence, and behavioural adequacy. It highlights that restricting similarity measures to a subset of tasks can shift rankings, and shows that a perfect postcondition score can still miss critical input coverage, underscoring the need for separate feedback on input and output constraints.
arXiv:2606. 23533v2 Announce Type: replace Abstract: Recent large language models (LLMs) are good at general text generation, but it is still hard to use them for domain-specific data generation because the output must follow strict formatting and structural rules.