The paper introduces Functional Entropy, a new uncertainty quantification technique for assessing the functional correctness of code generated by large language models. It evaluates token‑probability and sampling‑based methods across three programming languages and five LLMs, finding that token‑probability approaches generalize well while NLI‑based sampling fails due to semantic clustering. Functional equivalence methods, which replace NLI with an LLM‑based functional assessment, achieve superior AUROC and calibration in most model‑benchmark combinations.
By Dylan Bouchard, Mohit Singh Chauhan, Zeya Ahmad, Ho-Kyeong Ra
arXiv:2608. 14659v1 Announce Type: new Abstract: Large language models for code generation often produce incorrect solutions without reliable indicators of failure.
By Pranav Rakasi, Maanas Lalwani, Arnav Srivastava, Arya Palanivel, Tinuade Adeleke, Ruizhe Li, Sean Wu
arXiv:2608. 14653v1 Announce Type: cross Abstract: Prediction uncertainty is a widely adopted metric for quantifying model confidence, with downstream applications spanning model explanation, data selection, and prediction rollback.
By Xianzong Wu, Xiaohong Li, Yuejun Guo, Xinyang Liu, Tianlin Li, Junjie Wang, Qiang Hu
arXiv:2607. 12273v1 Announce Type: cross Abstract: As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences.
By Xiaoning Ren, Yinxing Xue, Lei Ma, Yuheng Huang
As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing.
The thesis explores Introspective Uncertainty Estimation (IUE) for large language models (LLMs) in code generation, aiming to determine whether hidden-state representations can indicate functional correctness at both response and line levels. Using LiveCodeBench and BigCodeBench, the study finds that hidden states provide a strong signal for overall correctness, with static single-token probes performing best, while dynamic strategies offer no consistent advantage. Although line-level fault localization is more challenging, a conditional Top‑K ranking approach remains effective, suggesting a two‑stage workflow that first screens responses for risk and then prioritizes line‑level checks.
By Thomas Klassert
arXiv:2508. 16131v3 Announce Type: replace-cross Abstract: Code completion entails the task of providing missing tokens given a surrounding context.
By Zoe Kotti, Konstantina Dritsa, Diomidis Spinellis, Panos Louridas
arXiv:2608. 08266v1 Announce Type: cross Abstract: Code generated by modern language models often reads naturally.
By Francisco Ribeiro, Sohaila Abdulsattar, Renata Gonzalez, Mahmoud Kassem, Sarah Nadi
arXiv:2511. 20709v2 Announce Type: replace-cross Abstract: Large language models (LLMs) and LLM-based coding agents are now used to generate code from natural-language specifications, yet ensuring such code is both functionally correct and secure remains a challenge.
By Rupam Patir, Keyan Guo, Suvadra Barua, Abhijeet Pathak, Dinesh Gudimetla, Jiawei Guo, Hongxin Hu, Haipeng Cai
arXiv:2608. 04439v1 Announce Type: cross Abstract: Large language models (LLMs) have made notable progress in code generation, but they still struggle on challenging tasks that require sophisticated algorithms or complex implementations.
By Yiru Dong, Richong Zhang, Fanshuang Kong, Si Chen
arXiv:2606. 19868v1 Announce Type: new Abstract: Although large language models (LLMs) have shown strong capabilities across a wide range of tasks, their outputs often remain unreliable and may contain hallucinations, making uncertainty estimation (UE) essential for building trustworthy LLMs.
By Jiayi Wang, Xu-Yao Zhang
arXiv:2606. 15589v1 Announce Type: cross Abstract: For tool-augmented language models, comparing natural-language reasoning with code-execution pipelines is difficult because the comparison changes both the intermediate representation and the execution mechanism.
By Terry Tong, Yu Feng, Surbhi Goel, Dan Roth