arXiv AI
Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 2
Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models
The paper investigates whether code large language models (CodeLLMs) inadvertently reproduce proprietary or sensitive code by evaluating seven state‑of‑the‑art training data detection (TDD) methods on eight CodeLLMs. It introduces CodeSnitch, a benchmark of 9,000 function‑level code samples across three languages, each labeled as included or excluded from training data, and applies mutation strategies based on the Type‑1 to Type‑4 code clone taxonomy to test TDD robustness. The study offers a systematic assessment of current TDD techniques for code and suggests directions for developing more effective detection methods.
Hugging Face Trending Papers
Jul 16
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored.
arXiv AI
Jul 17
Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality
arXiv:2607. 14816v1 Announce Type: cross Abstract: Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias.
arXiv Machine Learning
Sep 23
On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Quality
arXiv:2609.26388v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have made them widely used for code-related tasks. Identifier names are statistically informative in...
arXiv AI
Sep 2
Predicting Program Exit Code with LLMs and Programming Language Semantics
The paper introduces Program Executability Prediction (PrEx), a task that asks large language models (LLMs) to determine whether a program is semantically valid or invalid and, if invalid, to identify the violated formal rule. To evaluate this, the authors create a dataset of systematically generated invalid programs derived from valid ones and test open‑source coding LLMs across different semantic formalisms, semantic shifts, and program splits (human‑written, LLM‑translated, fuzzer‑generated). Results show that LLMs rely more on pre‑training priors than on the provided semantics, performing poorly on modified semantics and with increasing program complexity.
arXiv Machine Learning
5d ago
LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking
The paper introduces CETBench, a benchmark for assessing large language models (LLMs) on code-equivalence checking by generating program pairs through random code transformations. It demonstrates that even simple transformations significantly degrade LLM performance, especially in cross-lingual scenarios, and proposes a fine-tuning approach to improve results. The study provides detailed ablations on program difficulty and transformation types, offering insights into LLMs’ limited semantic understanding of code.