arXiv Computation and Language
When Does Better Confidence Improve a Knowledge-Graph Audit?
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Oct 16
Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence
The paper introduces RADAR, a framework for repairing and auditing confidence scores in large language models (LLMs). It separates confidence repair from deployment decisions and from error auditing, proposing Probe‑Anchored Isotonic Calibration (PAIC) for monotone mapping and a prompt‑clustered kernel test for detecting residual errors. Experiments across checkpoints and benchmarks show that richer predictors can improve confidence estimation, but projecting back to a scalar score can lose gains, and lower prediction risk does not always lead to better review decisions.
arXiv AI
Sep 25
Sequential knowledge editing breaks a model's ability to tell good evidence from bad, without costing it accuracy
The paper investigates how sequential knowledge editing can degrade a language model’s ability to discern reliable evidence from unreliable evidence without affecting overall accuracy. Using a conservatively tuned LoRA on Qwen2.5‑7B‑Instruct, the authors show that after 1,000 edits the model’s arbitration score for untouched facts drops by 36%, leading to higher error rates on its most confident decisions, while MMLU accuracy remains unchanged. The study also finds that in some model‑method combinations, sequential edits can reduce MMLU to chance levels even though edit success and locality remain perfect.
arXiv AI
Jul 7
When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions
arXiv:2607. 03386v1 Announce Type: new Abstract: Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable.
arXiv Statistics ML
Oct 27
Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences
arXiv:2610.33262v1 Announce Type: cross Abstract: We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity-credit programs in which expert verificatio...
arXiv Computation and Language
Oct 12
AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
AMLworld-Compact introduces an importance‑weighted downsampling protocol for evaluating large language models on AMLworld’s test splits, retaining all illicit transactions while sampling benign ones from supervised‑score strata. The HT‑Coreset method reduces evaluation targets by 271× on HI‑Small and 611× on LI‑Small, providing unbiased confusion‑count estimates with exact recall and ratio estimates for precision and F1. Experiments with boosted‑tree strata and Graph Feature Preprocessor features show lower F1 RMSE compared to uniform sampling, and audit tools reveal that lexical checks do not reliably indicate correctness of model outputs.
arXiv Machine Learning
Oct 22
Big Claims, Small Margins: Do Reported GraphLLM Gains Survive Simple Baselines?
The paper evaluates whether recent LLM-based node classification methods on text‑attributed graphs truly outperform simple baselines. By tuning four baseline models on the same data splits and comparing them to the best reported LLM results, the authors find that while some gains are modest (e.g., Correct & Smooth trailing by 2.3–2.8 percentage points), several reported improvements remain significant. They also highlight evaluation risks such as test‑set hyperparameter tuning, misuse of test‑pool labels, undisclosed baseline reuse, and inconsistencies across papers.