The paper introduces a runtime verification framework that monitors spoken air‑traffic control procedures by parsing controller‑pilot radio exchanges, integrating surveillance and onboard observations, and evaluating them against ICAO‑derived temporal obligations. The system generates time‑stamped traces, detects violations, and reports breached obligations with supporting observations. Evaluations on real traffic and synthetic scenarios show high accuracy (F1 = 0.85) and correct identification of procedural deviations in historical accident reconstructions.
arXiv:2608.24621v2 Announce Type: replace Abstract: Can language models be trusted in safety- critical operations? In such settings, strong per- formance on semantic metrics does not guaran- tee oper...
arXiv:2608. 19299v1 Announce Type: new Abstract: Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management have been semi-automated.
FLY-EVAL++ is an evidence-driven evaluation protocol designed for safety-constrained flight prediction with large language models. It combines deterministic verification of protocol compliance, physical feasibility, and safety constraints with rubric-guided aggregation into interpretable multi-dimensional scores. Applied to Flight Trajectory and Attitude Prediction, the protocol revealed that safety compliance is the most discriminative dimension among 66 LLMs, with models showing up to 28-point differences in safety scores and recurrent failures such as safety violations under physically plausible predictions and instability in multi-step rollouts.
arXiv:2608. 16349v1 Announce Type: new Abstract: Large language model (LLM) agents may assist flight crews with complex decisions and task execution, but existing aviation evaluations centered on static knowledge do not support systematic testing of procedural execution and safety compliance in interactive environments.
Large language model (LLM) agents may assist flight crews with complex decisions and task execution, but existing aviation evaluations centered on static knowledge do not support systematic testing of...
VeriPhy is an auditable physical‑verification system that transforms a text prompt into typed physical obligations and a statically validated execution plan before any video frames are generated. During execution, it gates calls to frozen low‑level experts (segmentation, tracking, counting, depth, OCR, audio‑event detection, etc.) and records provenance‑carrying evidence for each action. The system maps these records to a three‑valued state—supported, contradicted, or unknown—providing traceable verdicts that can be used to refine generation models.
arXiv:2608.21363v1 Announce Type: new Abstract: A protocol is presented for recording the governance decisions of automated AI runtimes. When a runtime releases, blocks, defers, redacts, or escalates...
FLY-EVAL++ is an evidence-driven evaluation protocol designed for safety-constrained flight prediction with large language models. It combines deterministic verification of protocol compliance, physical feasibility, and safety constraints, then aggregates results into interpretable multi-dimensional scores. Applied to Flight Trajectory and Attitude Prediction, the protocol reveals that safety compliance is the most discriminative metric, with models of similar predictive accuracy differing by over 28 points in safety score and exhibiting recurrent safety violations and instability in multi-step rollouts.
The paper introduces JevAct, a runtime verification framework that intercepts selected actions of large language model (LLM) agents before execution. JevAct compiles domain policies into executable state checks and uses a prefill-only decision model to provide probability‑based judgments, enabling feedback‑driven correction without modifying the underlying agent. Evaluations on eleven LLM agents across Airline, Telecom, and BFCL benchmarks show that JevAct improves agent reliability, especially on Telecom and BFCL tasks.
The paper introduces a runtime assurance layer for language‑model agents operating in operational workflows, addressing the evaluation–execution gap where approvals may become invalid after changes in facts, evidence, policy, or authority. The proposed state‑bound configuration separates probabilistic measurement from deterministic adjudication, binds execution allowances to the basis of an approval, invalidates them upon logged changes, and rechecks admissibility at dispatch. In a scripted 42‑case replay, the approach identified 15 later‑invalidated approvals that a non‑re‑evaluating comparator missed, holding 14 affected actions before dispatch and compensating one landed effect.
VeriPhy is an auditable physical‑verification system that evaluates generated video by compiling prompts into typed physical obligations and a statically validated execution plan before any frames are observed. During execution, it gates calls to frozen low‑level experts (e.g., segmentation, tracking, counting, depth, OCR, audio‑event detection) and returns provenance‑carrying evidence records, which are mapped to a three‑valued state (supported, contradicted, unknown) with full traceability. On a 1,500‑clip corpus of human‑annotated flaw records, VeriPhy accounts for 228 failures out of 304, outperforming a published question‑decomposition evaluator that accounts for 164, while also providing auditable evidence for each verdict.
arXiv:2609.13552v1 Announce Type: new Abstract: Generative AI is increasingly being used informally in Air Traffic Management (ATM) for tasks such as flight plan generation, trajectory interpretation...