The article "Bug Detection Blind Spots in AI Coding Harnesses (GStack and Beyond)" reports on 28 debugging experiments that show AI coding tools struggle more with missing information than with code complexity. It highlights that these tools exhibit blind spots when key details are absent, affecting their debugging performance.
A practical tutorial for recording model tool requests, real function results, patches, checks, screenshots, and a saved run log. The post How to Debug AI Coding Agents When They Change the Wrong Thing appeared first on Towards Data Science .
FuzzingBrain‑Bench V1 is a new benchmark that tests large language models (LLMs) on their ability to discover software bugs in open‑source projects. Unlike prior benchmarks that focus on a single target vulnerability, this benchmark gives models a Docker‑based harness and asks them to generate inputs that trigger as many distinct crashes as possible. The first version contains 77 challenges from 43 projects (36 C, 32 C++, 9 Java/JVM) and evaluates Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8, with Claude Opus 4.8 achieving the highest score by triggering crashes in 60 of 77 challenges.
A new analysis from OpenAI reveals issues in SWE-Bench Pro, a popular coding benchmark, raising concerns about reliability and accuracy in evaluating AI models.
The paper investigates how Large Language Models (LLMs) handle bug fixing compared to human-written patches by analyzing about 3,000 Codeforces submissions. It finds that LLMs often modify more lines than necessary and sometimes produce entirely new solutions, and that they solve more problems correctly when generating solutions from scratch rather than patching existing code. The study highlights implications for AI‑assisted programming tools, suggesting a shift toward incremental problem‑solving strategies.
The paper investigates how large language models (LLMs) handle bug fixing versus problem solving in competitive programming. Using a dataset of ~3,000 Codeforces submissions and their human fixes, the authors compare LLM-generated patches to human patches and assess whether LLMs prefer to modify buggy code or generate new solutions. Results show that LLMs often alter more lines than necessary and sometimes produce entirely new solutions, performing better when allowed to generate solutions from scratch rather than patching existing code.
arXiv:2609.37864v1 Announce Type: cross Abstract: Agent harness bugs exhibit unique characteristics and remain challenging for state-of-the-art software agents to repair. Progress in this area is fur...
arXiv:2608. 06153v1 Announce Type: cross Abstract: Automated skill evolution enables Large Language Model (LLM) agents to continuously improve without expensive retraining.
If you are a programmer and you don't feel "special" anymore, you are not alone The post The Era of No-Code AI: What You Need to Know appeared first on Towards Data Science .
arXiv:2609.12708v2 Announce Type: replace-cross Abstract: AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whe...
arXiv:2608. 04682v1 Announce Type: cross Abstract: Coding agents powered by large language models (LLMs) are increasingly adopted in software engineering (SWE) scenarios, capable of fixing a specific bug in large-scale codebase.
arXiv:2608. 19799v1 Announce Type: new Abstract: Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions.
arXiv:2606. 03852v1 Announce Type: cross Abstract: Large language models often generate code with bugs.