The paper studies how to assess risk in intent-based networking by examining uncertainty in large language model (LLM) translations of high‑level intents into network configurations. It introduces two uncertainty signals: sampling‑based predictive uncertainty for ranking translations by risk, and token‑level entropy for locating ambiguous source tokens. Experiments on a Llama‑3.1‑8B‑Instruct model fine‑tuned for Juniper EX3300 show that predictive uncertainty can rank risky translations, while token entropy correlates with ambiguity from parameters and keywords.
arXiv:2603. 24929v2 Announce Type: replace Abstract: Understanding and quantifying uncertainty in large language model (LLM) outputs is critical for reliable deployment.
Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods.
The paper challenges the common practice of estimating aleatoric uncertainty in large language models (LLMs) by generating multiple clarified inputs and comparing the resulting answers. It argues that answers are unnecessary, costly, and can introduce epistemic leakage, proposing instead a clarification-only method that directly assesses ambiguity from the space of plausible interpretations. Experiments on three benchmarks show the new approach improves AUROC, reduces computational cost, and yields uncertainty estimates less correlated with epistemic uncertainty.
Intent Engine is a natural‑language intent translation architecture that converts user intents into validated Service‑level Objectives (SLOs) for compute‑continuum microservice placement. It combines schema‑constrained extraction, retrieval‑grounded value construction from monitored infrastructure, and validation against supported constraints to produce reliable SLO artifacts. In evaluations on a 716‑record dataset, Intent Engine outperformed prompting baselines and a rule‑based parser, achieving a 0.941 total F1 score with GPT‑4.1 mini and reducing downstream placement failures from 30.8% to 2.1%.
The paper introduces a perturbation-based framework for detecting input ambiguity in large language models (LLMs) before attempting to disambiguate. It presents two detectors—a black-box rule-based perturbation library and a white-box importance-guided method that leverages internal model signals—both designed to preserve the original input’s label while remaining close to it. Experiments on open-domain, conversational, and visual QA datasets show these methods outperform existing output-side uncertainty and LLM-paraphrase-based baselines, underscoring the value of early ambiguity detection.
The study investigates whether lightweight open LLMs can reliably translate operator-stated intents into enforceable service-level objectives (SLOs) for intent-based networking (IBN). Ten LLMs with fewer than 10 B parameters were evaluated across five dimensions—translation accuracy, intent validation, hallucination, energy usage, and robustness to phrasing—using 716 placement intents. The best configurations achieved a strict F1 of 0.943, rejected 191 of 195 invalid intents, and incurred 0.044–0.148 Wh of GPU energy per translation, while realistic rephrasing reduced invalid intent rejection in most configurations.
arXiv:2603. 25450v2 Announce Type: replace Abstract: Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment.
arXiv:2603. 24967v2 Announce Type: replace Abstract: Understanding why a large language model (LLM) is uncertain about the response is important for their reliable deployment.
Large language models (LLMs) often answer ambiguous inputs as if they had a single intended meaning. This is problematic because input ambiguity is an upstream source of uncertainty: when a prompt adm...
LOGIC is a benchmark and evaluation framework that tests how language models can ground engineering requests in a deterministic inventory of candidate changes before propagating selected changes through an electrical traceability graph. The benchmark includes 168 scenarios—144 for selection and 24 for abstention—and evaluates three 7–8B models against intent‑agnostic, lexical, and structured‑evidence methods. Results show that structured evidence can achieve perfect candidate F1 on anchored cases, while large language models perform better on relational‑paraphrase cases; however, grounding accuracy drops as candidate inventories grow, and strict evidence gating reduces false positives but may also remove correct selections.
The paper introduces Label-Confidence-Aware Uncertainty Quantification (LCA-UQ), a method that uses Pointwise Kullback-Leibler divergence to align global entropy from multiple stochastic samples with the local confidence of a candidate answer. By bridging this gap, LCA-UQ improves the reliability and stability of uncertainty assessments in natural language generation. Experiments on popular LLMs and NLP datasets show that label sources significantly influence classification and that LCA-UQ outperforms existing uncertainty estimation approaches.
arXiv:2607. 03882v1 Announce Type: cross Abstract: LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language.