arXiv AI

Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference

arXiv:2606. 02430v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly integrated into high-performance computing (HPC) workflows, accelerating scientific discovery through diverse perspectives such as code generation and domain-specific decision-making.

arXiv AI

Jul 2

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

arXiv:2607. 00700v1 Announce Type: cross Abstract: LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging.

By Zhao Tian, Yingquan Zhao, Chenyao Suo, Meng Wang, Junjie Chen

llms agents benchmarks

arXiv AI

Jul 14

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.

By Pavel Adamenko, Mikhail Ivanov, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitry Babaev, Alena Fenogenova, Valentin Malykh

llms agents benchmarks

arXiv AI

Aug 11

PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.

By Dongjie Xu, Julius, Hanchi Dong, Minghua Tang, Yuxuan Sun, Ziwei Nie, Zicheng Liu, Dujun Qing, Jiajie Xu

llms agents benchmarks safety

arXiv AI

6d ago

Program Semantic Inequivalence Game with Large Language Models

arXiv:2505. 03818v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) can achieve strong performance on everyday coding tasks, but they can fail on complex tasks that require non-trivial reasoning about program semantics.

By Antonio Valerio Miceli-Barone, Vaishak Belle, Ali Payani

llms agents fine-tuning benchmarks

arXiv AI

Jun 24

VeriPilot: An LLM-Powered Verilog Debugging Framework

arXiv:2606. 23759v1 Announce Type: cross Abstract: Verilog debugging remains one of the most time-consuming stages in digital circuit design.

By Yihan Wang, Cheng Liu, Jiazheng Zhang, Lei Zhang, Long Cheng, Xiaowei Li, Huawei Li

llms benchmarks

arXiv AI

Jun 30

Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking

arXiv:2606. 29088v1 Announce Type: cross Abstract: There are various benchmarks to evaluate bugfixing capabilities of Large Language Models.

By Bal\'azs Szalontai, \'Abel Szauter, Bal\'azs M\'arton, P\'eter Verebics, Bal\'azs Pint\'er, Tibor Gregorics

llms fine-tuning benchmarks

arXiv AI

Jul 7

MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems

arXiv:2605. 28732v2 Announce Type: replace-cross Abstract: Memory is essential for enabling large language models to support long-horizon reasoning, yet existing memory systems remain unreliable and difficult to debug.

By Xinle Deng, Ruobin Zhong, Hujin Peng, Xiaoben Lu, Yanzhe Wu, Guang Li, Buqiang Xu, Yunzhi Yao, Jizhan Fang, Haoliang Cao, Junjie Guo, Yuan Yuan, Ziqing Ma, Yuanqiang Yu, Rui Hu, Baohua Dong, Hangcheng Zhu, Ningyu Zhang

llms rag benchmarks

arXiv AI

Jun 4

Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models

arXiv:2606. 04739v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generation (RAG) settings.

By Sabrina Kaniewski, Fabian Schmidt, Tobias Heer

llms rag

arXiv AI

Jun 3

Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks

arXiv:2606. 03606v1 Announce Type: cross Abstract: Large language models achieve strong performance on arithmetic reasoning benchmarks, and one common response to arithmetic brittleness is to delegate computation to code.

By Malia Barker, Bishal Lakha, Edoardo Serra, Francesco Gullo

llms benchmarks

arXiv AI

Aug 11

On the Robustness of LLMs' Internal Representation of Code Correctness

arXiv:2608. 08266v1 Announce Type: cross Abstract: Code generated by modern language models often reads naturally.

By Francisco Ribeiro, Sohaila Abdulsattar, Renata Gonzalez, Mahmoud Kassem, Sarah Nadi

llms

arXiv AI

Jul 7

Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL

arXiv:2607. 03833v1 Announce Type: cross Abstract: While Large Language Models (LLMs) have achieved remarkable success in Text-to-SQL tasks, their deployment in real-world environments is hindered by latent reliability issues.

By Hanqing Wang, Yongdong Chi, Jian Yang, Lei Yang, Jiehui Zhao, Yun Chen, Guanhua Chen

llms fine-tuning benchmarks data-engineering

arXiv AI

Jun 9

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.

By Zenghui Zhou, Man Li, Xiaoke Fang, Xinyi Zhou, Weibin Lin, Zheng Zheng

llms benchmarks

Related stories