arXiv AI
Jul 24
Code Monitor Red Teaming for Public-Test-Passing Code
arXiv:2607. 20852v1 Announce Type: new Abstract: Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness.
Code Monitor Red Teaming for Public-Test-Passing Code
Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness. We study a deployment-like monitoring problem: after code has passed public tests, can a weaker LLM verifier identify the residual hidden bugs?
Related stories
arXiv AI
Jun 24
AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability
arXiv:2606. 24589v1 Announce Type: new Abstract: Scaling adversarial evaluation of large language models requires both a method for generating hard inputs and a reliable way to confirm that resulting failures are real.
arXiv AI
Jul 28
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
arXiv:2607. 22883v1 Announce Type: cross Abstract: While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code.
arXiv AI
Jun 3
FLARE: Fine-Grained Diagnostic Feedback for LLM Code Refinement
arXiv:2606. 03852v1 Announce Type: cross Abstract: Large language models often generate code with bugs.
arXiv AI
Jul 28
Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)
arXiv:2607. 22880v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation.
Hugging Face Trending Papers
Jul 13
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
Production LLM agents such as Claude Code and Codex operate over untrusted content, files, commands, and workspace state, making safety failures directly actionable. Red-teaming must therefore keep pace with evolving models and tools.
arXiv AI
Jul 14
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
arXiv:2607. 11698v1 Announce Type: cross Abstract: Production LLM agents such as Claude Code and Codex operate over untrusted content, files, commands, and workspace state, making safety failures directly actionable.
Hugging Face Trending Papers
Jul 24
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code. This paper presents a new metric to quantitatively measure the "misguidance effect," a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it.
arXiv AI
Jul 7
JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode
arXiv:2607. 02825v1 Announce Type: cross Abstract: We release \textsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection.
Hugging Face Trending Papers
Jun 24
Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study
Software vulnerability remediation is a cognitively demanding task that requires specialized security expertise often lacking in general developers. In the meantime, Large Language Models (LLMs) assisted tools show potential in vulnerability detection, location, and repair tasks.
arXiv AI
Jul 23
Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes
arXiv:2607. 19843v1 Announce Type: cross Abstract: Large language models (LLMs) have made automated program repair (APR) increasingly practical for real-world bugs, but repairing directly from bug reports remains underconstrained.
arXiv AI
Aug 12
REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems
arXiv:2608. 10669v1 Announce Type: new Abstract: Large language model (LLM) agents combine language-based reasoning with external tools to perform complex tasks.