The paper introduces Functional Entropy, a new uncertainty quantification technique for assessing the functional correctness of code generated by large language models. It evaluates token‑probability and sampling‑based methods across three programming languages and five LLMs, finding that token‑probability approaches generalize well while NLI‑based sampling fails due to semantic clustering. Functional equivalence methods, which replace NLI with an LLM‑based functional assessment, achieve superior AUROC and calibration in most model‑benchmark combinations.
By Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad, Ho-Kyeong Ra
arXiv:2608. 14659v1 Announce Type: new Abstract: Large language models for code generation often produce incorrect solutions without reliable indicators of failure.
By Pranav Rakasi, Maanas Lalwani, Arnav Srivastava, Arya Palanivel, Tinuade Adeleke, Ruizhe Li, Sean Wu
arXiv:2608. 14653v1 Announce Type: cross Abstract: Prediction uncertainty is a widely adopted metric for quantifying model confidence, with downstream applications spanning model explanation, data selection, and prediction rollback.
By Xianzong Wu, Xiaohong Li, Yuejun Guo, Xinyang Liu, Tianlin Li, Junjie Wang, Qiang Hu
arXiv:2607. 12273v1 Announce Type: cross Abstract: As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences.
By Xiaoning Ren, Yinxing Xue, Lei Ma, Yuheng Huang
As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing.
The thesis explores Introspective Uncertainty Estimation (IUE) for large language models (LLMs) in code generation, aiming to determine whether hidden-state representations can indicate functional correctness at both response and line levels. Using LiveCodeBench and BigCodeBench, the study finds that hidden states provide a strong signal for overall correctness, with static single-token probes performing best, while dynamic strategies offer no consistent advantage. Although line-level fault localization is more challenging, a conditional Top‑K ranking approach remains effective, suggesting a two‑stage workflow that first screens responses for risk and then prioritizes line‑level checks.
By Thomas Klassert