arXiv AI
Jul 7
Evaluating LLM-Based Regression Test Generation
arXiv:2501. 11086v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have shown tremendous promise in automated software engineering.
Causal Syntax Circuits in CodeLMs: Budget and Probe Decide What Looks Localised
The paper investigates whether the tendency of code language models to default to Python when no language is specified is driven by a small, identifiable set of internal components. Using a comprehensive pipeline—including ablation, activation patching, greedy circuit extraction, and significance testing—across 55 model-language configurations, the authors find that this preference is large, grows with model size, and persists after instruction tuning. They also show that the apparent localization of this behavior depends heavily on analysis choices such as component budget and probe design, and that interventions can sometimes alter the model’s output.
Related stories
arXiv Machine Learning
21h ago
Which Signals of Code Correctness Survive Distribution Shift? Hidden-State and Causal Predictors of Program and Verdict Correctness
arXiv:2611.07876v1 Announce Type: cross Abstract: Large language models increasingly judge code, yet without external feedback such judgments are unreliable, and verifying them by executing tests or...
arXiv Machine Learning
Sep 18
Measurement Under Selection: Decoy-Calibrated Failure Audits for Language Models
The paper introduces Janus, a method for validating error patterns in language models by comparing error rates across predefined yes/no properties and using shuffled decoy labels to set significance thresholds. Janus requires that a pattern’s error difference surpasses the decoy-derived threshold and is replicated on held‑out data before reporting. Experiments on a controlled code‑finding task confirm several meaningful error patterns, while on MuSiQue and LongBench v2 Janus reports no confirmed patterns for the tested properties, contrasting with standard shuffling tests that sometimes confirm patterns.
arXiv Machine Learning
Sep 23
Rewired or Gated? How Instruction Tuning Shapes Knowledge-Conflict Circuits in LLMs
arXiv:2609.25602v1 Announce Type: new Abstract: In language models, the choice between believing the prompt and believing the weights is made by a handful of identifiable attention heads. Instruction...
arXiv AI
Aug 11
MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models
arXiv:2603. 28590v3 Announce Type: replace Abstract: Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs.
arXiv AI
Jul 22
Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models
arXiv:2607. 19257v1 Announce Type: cross Abstract: Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade.
arXiv AI
Sep 23
Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code
arXiv:2608.28021v2 Announce Type: replace-cross Abstract: Large language models increasingly author Infrastructure-as-Code (IaC), where one insecure default is provisioned straight into production. P...
arXiv AI
Aug 26
Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
The study investigates why small language model agents tend to repeat a tool call that just failed. By recording the failed call and its error message in the transcript, the authors measure a negative corrective gain—agents are more likely to repeat the failed action, with a drop of about 1.03 nats per token. The problem is traced to the harness design rather than the model’s understanding of errors, and the authors show that replacing the verbatim call with a runtime-generated description of the failure can reduce this backfiring effect by 76%.
arXiv AI
Jun 8
SWE-IF: Aligning Code Evaluation with Human Preference
arXiv:2510. 07315v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check.
arXiv AI
Aug 19
Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations
The paper investigates whether the hidden activations of large language models (LLMs) contain signals about the vulnerability of C/C++ code when the code is provided as context. By extracting prefill token activations from four LLMs and training small MLP probes, the authors achieve an average F1 score of 41.7% across four benchmarks, with the best probe matching state‑of‑the‑art fine‑tuned classifiers on the Devign dataset. The results suggest that a coding LLM’s internal representation can inform vulnerability detection, opening the door to lightweight, model‑native screening methods.
arXiv Computation and Language
Oct 15
HERO: Harnessing Evidence for Test-time Refinement and Prompt Optimization in Program Generation
HERO is a method that improves large language models for program generation by combining offline and test‑time harness optimization. It learns reusable prompt updates from execution‑verified successes and failures, validates candidate revisions before adoption, and then refines each generated code instance using execution feedback. On four SQL, Cypher, and Python benchmarks, HERO achieves the best offline performance, raising pass@1 by 7.6 points over the initial prompt and adding another 2.0 points through test‑time refinement while improving execution efficiency.
arXiv AI
Aug 11
Contamination Means Overestimation? A Fine-Grained Empirical Study in Code Intelligence
arXiv:2506. 02791v4 Announce Type: replace-cross Abstract: In recent years, code intelligence has gained increasing importance in the field of automated software engineering.