arXiv AI By Zhaorui Yang, Haichao Zhu, Qian Zhang, Rajiv Gupta, Ashish Kundu

SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization

Read the original on arXiv AI →

SemLoc transforms free‑form reasoning from large language models into executable semantic constraints that are attached to specific code anchors such as variable definitions, branches, loops, or returns. These constraints are inserted as runtime checks, and the resulting semantic violation spectrum is used to guide counterfactual verification and patch generation. On the QuixBugs and new SemBugs datasets, SemLoc’s constraints achieve high ranking of faulty lines and a high proportion of verifiable, correct patches, matching the accuracy of the underlying LLMs while outperforming traditional spectrum‑based methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Sep 12

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

SemVerBench is a benchmark that evaluates how well large language models (LLMs) understand and apply version-constraint resolution semantics, such as determining whether a version satisfies constraints like ^1.2.3 or >=2.0. The study finds that many models struggle with certain corner cases, with GPT‑5.1 performing poorly while Claude and Opus perform much better. The authors suggest that the failures stem from an activation/application gap rather than a lack of knowledge, and recommend that coding agents delegate version resolution to a dedicated resolver tool.

By Qibai Chen, Zeming Liu
arXiv AI
Sep 24

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati
arXiv AI
Oct 9

Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes

The paper investigates whether large language models (LLMs) can automatically verify no-code bug fixes—solutions that instruct users to change settings, upgrade versions, or alter workflows—by executing them in a real browser environment. Using 322 fixes from a benchmark, the study tests three executor agents (OpenCUA‑72B, Claude Sonnet 5, and Meta’s Muse Glimmer) and finds that only 17.6% of issues could be set up and passed sanity checks, with resolution rates ranging from 14.6% to 49.7% depending on the executor. Even the best configuration resolved fewer than half of the fixes, and executor choice significantly affected outcomes, highlighting the need for execution-based verification before such fixes reach users.

By Utku Boran Torun, Veli Karakaya, Eray T\"uz\"un
arXiv AI
Sep 21

SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

arXiv:2609.21190v1 Announce Type: cross Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corr...

By George Ma, Benjamin Mikek, Haoyu Li, Ferhat Erata, Yuhao Zhang, Zeren Shui, Behrooz Omidvar Tehrani, Jun Huan, Murali Krishna Ramanathan, Somayeh Sojoudi, Hao Zhou, Anoop Deoras