arXiv AI September 24, 2026 By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

Read the original on arXiv AI →

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 20

Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios

arXiv:2604. 06742v2 Announce Type: replace-cross Abstract: The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch.

By Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, David Lo

llms agents benchmarks

arXiv AI

Jul 2

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

arXiv:2607. 00700v1 Announce Type: cross Abstract: LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging.

By Zhao Tian, Yingquan Zhao, Chenyao Suo, Meng Wang, Junjie Chen

llms agents benchmarks

arXiv AI

Jul 14

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

arXiv:2507. 11059v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Language Models (LLMs) in software engineering has revealed critical limitations in existing benchmarks, particularly the widely used SWE-bench dataset.

By Pavel Adamenko, Mikhail Ivanov, Aidar Valeev, Rodion Levichev, Pavel Zadorozhny, Ivan Lopatin, Dmitry Babaev, Alena Fenogenova, Valentin Malykh

llms agents benchmarks

arXiv AI

Aug 3

RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment

arXiv:2507. 22580v2 Announce Type: replace-cross Abstract: Automated Program Repair (APR) seeks to automatically correct software bugs without requiring human intervention.

By Marcos Fuster-Pena, David de-Fitero-Dominguez, Antonio Garcia-Cabot, Eva Garcia-Lopez

llms reinforcement-learning benchmarks

arXiv AI

Jul 14

TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation

arXiv:2509. 24148v3 Announce Type: replace-cross Abstract: Test-Driven Development (TDD) is a widely adopted practice that requires developers to create and execute tests alongside implementation.

By Yiran Hu, Nan Jiang, Shanchao Liang, Yi Wu, Lin Tan

llms agents

arXiv AI

Aug 13

Program Semantic Inequivalence Game with Large Language Models

arXiv:2505. 03818v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) can achieve strong performance on everyday coding tasks, but they can fail on complex tasks that require non-trivial reasoning about program semantics.

By Antonio Valerio Miceli-Barone, Vaishak Belle, Ali Payani

llms agents fine-tuning benchmarks

One email a morning, machine-written

Related stories

Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios

LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment

TENET: One Step Toward Test-Driven Development for Repository-Level Code Generation

Program Semantic Inequivalence Game with Large Language Models