arXiv AI
Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimization
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 24
Not All Invariants Are Equal: Curating Training Data to Accelerate Program Verification with SLMs
arXiv:2603. 15510v2 Announce Type: replace Abstract: The synthesis of inductive loop invariants remains a critical bottleneck in automated program verification.
arXiv Machine Learning
Jul 1
An Empirical Study of Security Calibration in Large Language Models for Code
arXiv:2606. 31159v1 Announce Type: cross Abstract: Large Language Models (LLMs) are rapidly transforming software development, yet their use in security-critical contexts raises a key question: do models know when their generated code is insecure?
arXiv AI
Aug 18
T-LLM Compiler: Trusted LLM-based Code Optimization and Verification Framework
arXiv:2608. 14953v1 Announce Type: new Abstract: Recent advances in Large Language Models (LLMs) have opened opportunities to apply high-level code transformations to the field of code optimization, and it has since emerged as one of the most fundamental tasks for LLMs to perform; however, at present, LLMs struggle to apply wide-ranging code optimization tasks due to both the complexity of the code and the inability to independently verify the correctness of the transformations.
arXiv AI
Oct 9
Characterizing Overconfident Failure in LLM-Based Code Generation
The paper investigates how large language models (LLMs) used for code generation can produce syntactically plausible but incorrect programs, a phenomenon termed overconfident failure. By evaluating four open-source code models across three execution-based benchmarks, the authors examine whether token-level uncertainty metrics can predict execution correctness, analyze overconfidence at both global and local levels, and test mitigation strategies such as selective generation and instruction tuning. Their findings show that existing uncertainty signals are only partially reliable, overconfidence persists across models, instruction tuning can increase certainty without improving correctness discrimination, and common mitigation techniques do not fully resolve the issue. "whyItMatters":"The study highlights that current uncertainty metrics and mitigation strategies are insufficient to reliably detect or prevent incorrect code generation, underscoring the need for better reliability signals in LLM-based code synthesis."
arXiv AI
Oct 27
Evaluating Harmful Memorization in LLM Code Generation
The paper introduces the Harmful Memorization Index (HMI) to measure when large language models (LLMs) fail to adapt to small semantic changes in coding tasks, producing code that is superficially similar to the original solution. It proposes an automated code‑rewriting pipeline that generates semantically different variants of tasks by altering core logic elements, and uses HMI to combine similarity metrics with performance degradation. Experiments on MBPP+ and BigCodeBench show that HMI does not simply grow with model size, is higher for certain logic‑edit types, and that supervised fine‑tuning can increase HMI while PPO offers a better accuracy–HMI trade‑off.
arXiv AI
Sep 2
Predicting Program Exit Code with LLMs and Programming Language Semantics
The paper introduces Program Executability Prediction (PrEx), a task that asks large language models (LLMs) to determine whether a program is semantically valid or invalid and, if invalid, to identify the violated formal rule. To evaluate this, the authors create a dataset of systematically generated invalid programs derived from valid ones and test open‑source coding LLMs across different semantic formalisms, semantic shifts, and program splits (human‑written, LLM‑translated, fuzzer‑generated). Results show that LLMs rely more on pre‑training priors than on the provided semantics, performing poorly on modified semantics and with increasing program complexity.