arXiv AI
Oct 2

Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models

The paper investigates whether code large language models (CodeLLMs) inadvertently reproduce proprietary or sensitive code by evaluating seven state‑of‑the‑art training data detection (TDD) methods on eight CodeLLMs. It introduces CodeSnitch, a benchmark of 9,000 function‑level code samples across three languages, each labeled as included or excluded from training data, and applies mutation strategies based on the Type‑1 to Type‑4 code clone taxonomy to test TDD robustness. The study offers a systematic assessment of current TDD techniques for code and suggests directions for developing more effective detection methods.

By Tianlin Li, Yunxiang Wei, Zhiming Li, Aishan Liu, Qing Guo, Xianglong Liu, Dongning Sun, Yang Liu
arXiv Machine Learning
Sep 16

Type-IV Code Clone Detection via Layer-Wise Non-Contrastive Representation Learning

The paper introduces LWVIC4Code, a non‑contrastive, layer‑wise representation learning method for detecting Type‑IV code clones—semantically equivalent fragments that differ syntactically. It builds on the VICReg framework, adding cross‑layer consistency regularization and depth‑dependent weighting to refine semantic information across transformer layers. Experiments on Python and multi‑language datasets show that LWVIC4Code matches or outperforms contrastive baselines and zero‑shot large language models, generalizing well to Java and C# without requiring negative samples.

By Luciano Marchezan, Kevin Delcourt, Eugene Syriani, Houari Sahraoui
arXiv AI
Oct 1

EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses

EngramBench is a new benchmark designed to evaluate skill evolution in autonomous agents by focusing on genuine capability abstraction rather than solution copying. It includes 30 learning tasks and 13 unseen transfer tasks that require agents to manage complex, multi-hour development cycles with LLM‑simulated users. The study shows that while static skill banks cannot eliminate the need for precise code implementation, they effectively reduce redundant context and cut overall coding time by more than 55%.

By Zhixuan Tan, Pengjie Gu, Zhao Li, Yihan Hu, Xu He, Dong Li, Jianye Hao