arXiv:2601. 17717v3 Announce Type: replace Abstract: Large Language Models (LLMs) have emerged as powerful tools for generating data across various modalities.
By Kaituo Zhang, Mingzhi Hu, Hoang Anh Duy Le, Fariha Kabir Torsha, Zhimeng Jiang, Minh Khai Bui, Chia-Yuan Chang, Yu-Neng Chuang, Zhen Xiong, Ying Lin, Guanchu Wang, Na Zou
The paper introduces LeDQeR, an approach that uses large language models to automatically generate data quality rules for rule‑based enterprise tools. It follows a generate‑filter framework where the LLM proposes candidate rules from dirty data, and four filters ensure the rules are executable, correct, generalizable, and non‑redundant. Experiments show that LeDQeR produces effective, compact rule sets across diverse datasets and error types.
By Anna-Christina Glock, Thomas H\"utter, Johannes F\"urnkranz, Wolfram W\"o{\ss}, Christine Dominka-Kiss, Lisa Ehrlinger
arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.
By Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella Biderman
Litmus is a zero-label system that automatically designs evaluation and monitoring metrics for AI pipelines by extracting evaluation intent from source code and targeted interrogation. It identifies what must be measured and why, then generates a justified, per-stage metric portfolio. In experiments on three real code-defined pipelines, Litmus outperforms AutoMetrics and DynamicRubric baselines in concern coverage, stage span, redundancy, and validity against per-row quality labels, especially on scientific QA.
By Prajjwal Gupta, Prasang Gupta, Vishal Bhutani, Apoorva Sharma, Sumanth Chundru, Waqar Sarguroh, Kevin Paul
The paper introduces AXIOM, a perturbation-based framework for creating a balanced benchmark of code evaluation problems. AXIOM refines program quality by applying controlled semantic perturbations and then calibrates scores through selective manual annotation, resulting in the multilingual AXIOMBENCH dataset with 1,962 problems. This approach addresses limitations of prior benchmarks that rely on coarse binary labels or subjective human scoring.
By Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu