arXiv Machine Learning
SafeLLM4SE: Statistical Evaluation and Reporting for LLM-based Software Engineering Systems
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 15
Introspective Uncertainty Estimation for LLM-Based Code Generation
The thesis explores Introspective Uncertainty Estimation (IUE) for large language models (LLMs) in code generation, aiming to determine whether hidden-state representations can indicate functional correctness at both response and line levels. Using LiveCodeBench and BigCodeBench, the study finds that hidden states provide a strong signal for overall correctness, with static single-token probes performing best, while dynamic strategies offer no consistent advantage. Although line-level fault localization is more challenging, a conditional Top‑K ranking approach remains effective, suggesting a two‑stage workflow that first screens responses for risk and then prioritizes line‑level checks.
arXiv AI
Jun 2
Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
arXiv:2606. 00920v1 Announce Type: cross Abstract: Run-level pass rate overstates retry-free coverage by up to 17.
arXiv AI
Jul 7
Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study
arXiv:2607. 03174v1 Announce Type: cross Abstract: Software diversity has been extensively studied as a means of reducing the risk of common-mode failures.
arXiv AI
Aug 24
Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda
Large Language Models (LLMs) are evolving from simple code completion tools to repository‑scale agents capable of retrieving context, editing files, executing tools, and engaging in security‑sensitive workflows. A structured survey up to May 31 2026 reviews LLM work across software engineering and security tasks, adaptation mechanisms, artifact granularity, and evaluation design, and introduces an assurance framework that separates functional correctness, security, operational reliability, evidence provenance, and agent authority. The review highlights that while execution feedback and repository access improve engineering task completion, they do not guarantee security, and static‑analysis labels rarely ensure deployable correctness; it also identifies common validity threats and proposes a minimum reporting protocol and a research agenda focused on jointly secure‑and‑functional benchmarks, repository‑scale threat models, calibrated human oversight, longitudinal maintainability evidence, and reproducible agent evaluation.
arXiv AI
Aug 6
ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation
arXiv:2608. 04439v1 Announce Type: cross Abstract: Large language models (LLMs) have made notable progress in code generation, but they still struggle on challenging tasks that require sophisticated algorithms or complex implementations.
arXiv AI
Sep 16
Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision
The paper investigates how to improve confidence calibration for large language models (LLMs) used in automated code revision (ACR). It proposes applying local Platt-scaling to three fine-grained confidence scores, rather than the conventional global method, and demonstrates that this approach consistently reduces calibration error across multiple tasks, metrics, and model sizes. The study shows that fine-grained calibration, especially when combined with global scaling, yields more reliable confidence estimates for ACR tasks.