The study evaluates frontier large language models—Sol 6.1, Luna 6, and Qwen 3.8—in their ability to localize faults in LeetCode solutions without tools, with optional debugger assistance, or with encouraged debugger use. Across 610 seeded single‑line faults and five behavior‑preserving transformations, the agents achieved 98.2–99.9% accuracy on original code and 96.7–99.9% on transformed code when no tools were used. Encouraged debugger assistance further improved Luna 6 and Qwen 3.8’s accuracy by 61–63% of remaining errors, albeit at a 14.7–27.0× increase in token usage, while Sol 6.1 maintained 99.9% accuracy regardless of debugger use.
arXiv:2608. 14863v1 Announce Type: cross Abstract: LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified.
The paper investigates whether large language models (LLMs) can automatically verify no-code bug fixes—solutions that instruct users to change settings, upgrade versions, or alter workflows—by executing them in a real browser environment. Using 322 fixes from a benchmark, the study tests three executor agents (OpenCUA‑72B, Claude Sonnet 5, and Meta’s Muse Glimmer) and finds that only 17.6% of issues could be set up and passed sanity checks, with resolution rates ranging from 14.6% to 49.7% depending on the executor. Even the best configuration resolved fewer than half of the fixes, and executor choice significantly affected outcomes, highlighting the need for execution-based verification before such fixes reach users.
arXiv:2607. 18754v1 Announce Type: new Abstract: LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.
arXiv:2608. 14680v1 Announce Type: new Abstract: Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails.
arXiv:2608. 02643v1 Announce Type: cross Abstract: Computer-use agents (CUAs) operate real desktop and web interfaces through screenshots, mouse and keyboard actions, and stateful UI feedback, yet their failures remain difficult to diagnose and repair.
arXiv:2607. 28545v2 Announce Type: replace-cross Abstract: Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began.
arXiv:2504. 20412v3 Announce Type: replace-cross Abstract: Fuzzing frameworks like syzkaller have uncovered thousands of Linux kernel crashes, many of which are critical and security-sensitive.
arXiv:2607. 18161v1 Announce Type: cross Abstract: Coding agents are increasingly used to accelerate code generation in many downstream tasks, such as fixing bugs, building applications, and prototyping.
The paper examines how large‑language‑model (LLM) decompilers, which produce clean, idiomatic C code, are currently evaluated mainly on recompilability and passing shipped tests. It shows that these metrics can mask significant behavioral differences: a decompiled function may recompile and pass all tests yet diverge on other inputs or lose disclosed vulnerabilities. To address this, the authors propose Decompile‑Diverge, a behavioral oracle that synthesizes drivers, fuzzes inputs, and compares the decompiled code’s behavior to the original, revealing divergences in up to 13% of cases and exposing gaps in current evaluation suites.
SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/Codex CLI as the top solver and Claude Opus 4.8/mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.
Post-training quantization reduces the cost of deploying language-model agents, but its effect on recovery from temporary tool failures can depend on how recovery is evaluated. We compare 8-bit and 4-...
arXiv:2605. 05000v2 Announce Type: replace-cross Abstract: LLM agents have been increasingly adopted for solving security tasks.