arXiv AI
Jul 10
When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
arXiv:2607. 08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al.
Evidence-Grounded Ensemble Diagnosis of 802.11 Packet Captures: A Multi-Stage Pipeline with Deterministic Reliability Scoring
arXiv:2606. 06871v1 Announce Type: new Abstract: Diagnosing 802.
Related stories
arXiv AI
Aug 26
Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model
The paper audits a developer‑accessible on‑device language model, revealing that it can confidently produce incorrect answers while refusing benign prompts, a phenomenon termed task‑asymmetric miscalibration. The model’s confident outputs are surface‑indistinguishable, with classifiers based on user‑visible features failing to separate correct from wrong responses. The authors propose a model‑agnostic audit protocol, a surface‑indistinguishability test, and a black‑box consistency wrapper that improves reliability without requiring model access.
arXiv AI
Aug 11
READER: Dynamic LLM Provenance from Query-Varying Interactions
arXiv:2606. 10794v3 Announce Type: replace Abstract: Existing black-box LLM provenance methods achieve comparability by querying every candidate model with the same diagnostic prompts.
arXiv AI
2d ago
Counterfactual Fragility Certificates: Exposing High-Confidence Brittleness under Structured Evidence Failure
arXiv:2609.00366v1 Announce Type: cross Abstract: High test accuracy and good aggregate calibration do not show whether an individual prediction is structurally supported by its evidence. In tabular...
arXiv AI
Aug 3
EarlyDx: An Admission-Anchored Benchmark for Open-Ended Generation of Evidence-Supported ED-Encounter Diagnoses
arXiv:2607. 28788v1 Announce Type: new Abstract: Clinical diagnosis at hospital admission must be made rapidly from limited, incomplete evidence.
arXiv Machine Learning
Aug 4
Real-Time Detection and Repair of LLM Agent Failures
arXiv:2608. 02464v1 Announce Type: cross Abstract: LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself.
arXiv AI
2d ago
SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task
arXiv:2609.00654v1 Announce Type: new Abstract: We describe the SciTrue team's participation in both subtasks of the NTCIR-19 SciClaimEval task~\cite{sciclaimeval}, which asks systems to verify scien...
arXiv AI
3d ago
ASTRA - Agentic System for Ticket Resolution and Analysis
arXiv:2608.28790v1 Announce Type: cross Abstract: Technical operations teams resolve large volumes of incidents by synthesizing fragmented evidence from ticket text, historical cases, system logs, an...
arXiv Machine Learning
Jul 30
Same Evidence, Different Target: Decoding How Diagnostic Evidence Bears on Causal Questions from Language-Model States
arXiv:2607. 26929v1 Announce Type: cross Abstract: The same diagnostic result can support or challenge one causal claim yet fail to address another when the claims concern different populations, outcomes, estimands, pathways, or identifying assumptions.
arXiv AI
Aug 11
From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents
arXiv:2608. 05235v1 Announce Type: cross Abstract: Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings and retain the resulting trajectories to guide later decisions.
arXiv Computation and Language
Aug 25
Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
arXiv:2608.22483v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is mis...
arXiv AI
1d ago
ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations
The paper introduces ClaimReceipt, a specification and verifier that checks whether a claim in an agent evaluation can be recomputed from retained evidence (sufficiency) and whether the evidence covers the entire experiment set (coverage). Using the CR‑2 verifier on 1,392 historical records, the authors demonstrate accurate reproduction of audit verdicts, non‑redundant field groups, and zero false positives on semantic faults. In a prospective CR‑3 run, the system correctly flags missing receipts and preserves coverage when private evidence is withheld, while adding minimal overhead to inference time and transaction size.