arXiv AI
Jul 24
Code Monitor Red Teaming for Public-Test-Passing Code
arXiv:2607. 20852v1 Announce Type: new Abstract: Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness.
Code Monitor Red Teaming for Public-Test-Passing Code
Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness. We study a deployment-like monitoring problem: after code has passed public tests, can a weaker LLM verifier identify the residual hidden bugs?
Related stories
arXiv AI
Aug 19
Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations
The paper investigates whether the hidden activations of large language models (LLMs) contain signals about the vulnerability of C/C++ code when the code is provided as context. By extracting prefill token activations from four LLMs and training small MLP probes, the authors achieve an average F1 score of 41.7% across four benchmarks, with the best probe matching state‑of‑the‑art fine‑tuned classifiers on the Devign dataset. The results suggest that a coding LLM’s internal representation can inform vulnerability detection, opening the door to lightweight, model‑native screening methods.
arXiv AI
Jun 24
AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability
arXiv:2606. 24589v1 Announce Type: new Abstract: Scaling adversarial evaluation of large language models requires both a method for generating hard inputs and a reliable way to confirm that resulting failures are real.
arXiv AI
Jul 28
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
arXiv:2607. 22883v1 Announce Type: cross Abstract: While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code.
arXiv AI
4d ago
Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models
arXiv:2609.35841v1 Announce Type: cross Abstract: Mutation testing evaluates test-suite adequacy by injecting synthetic faults into program code. However, traditional rule-based tools often generate...
arXiv AI
Jun 3
FLARE: Fine-Grained Diagnostic Feedback for LLM Code Refinement
arXiv:2606. 03852v1 Announce Type: cross Abstract: Large language models often generate code with bugs.
arXiv AI
Jul 28
Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)
arXiv:2607. 22880v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation.
Hugging Face Trending Papers
Jul 13
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
Production LLM agents such as Claude Code and Codex operate over untrusted content, files, commands, and workspace state, making safety failures directly actionable. Red-teaming must therefore keep pace with evolving models and tools.
arXiv AI
Jul 14
Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming
arXiv:2607. 11698v1 Announce Type: cross Abstract: Production LLM agents such as Claude Code and Codex operate over untrusted content, files, commands, and workspace state, making safety failures directly actionable.
arXiv AI
Sep 24
CART: Closed-Loop Adaptive Red Teaming for Large Language Models
CART (Closed‑Loop Adaptive Red Teaming) is a framework that iteratively uses results from red‑teaming tests to guide subsequent probes, thereby expanding risk coverage and maintaining diversity. It separates the roles of Challenger (test generator), Target (model or agent under test), and Judge (result evaluator), enabling independent study of each component. Across multiple evaluation families, CART uncovers more failures and higher risk than static prompt replay, demonstrating that adaptive, continuous testing reveals weaknesses that fixed‑prompt methods miss.
Hugging Face Trending Papers
Jul 24
Evaluating and Mitigating the Misguidance Effect of Buggy Code in LLM-Generated Unit Tests
While Large Language Models (LLMs) show great promise for automating unit test generation, recent studies suggest that the quality of generated tests can be negatively impacted when models are prompted with buggy code. This paper presents a new metric to quantitatively measure the "misguidance effect," a phenomenon where buggy code steers LLMs toward generating tests that validate its erroneous behavior rather than expose it.
arXiv AI
Jul 7
JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode
arXiv:2607. 02825v1 Announce Type: cross Abstract: We release \textsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection.