The paper introduces VERDICT, an LLM-based agent that converts clinical trial matching tasks into SMT problems to ensure consistent policy application and accountable decisions. VERDICT outperforms other LLM-only and neurosymbolic baselines on accuracy, achieves perfect policy consistency, and generates clinician-preferred rationales grounded in explicit assumptions and pivotal conditions. It also demonstrates improved counterfactual self‑faithfulness, meaning changes in pivotal conditions appropriately alter decisions.
arXiv:2604. 14892v3 Announce Type: replace-cross Abstract: Evaluating medical AI systems using expert clinician panels is costly and slow, motivating the use of large language models (LLMs) as alternative adjudicators.
The study examines whether large language models truly base legal verdicts on the statutes or precedents they cite. By fixing case facts and swapping the cited authority with an unrelated one, researchers found that while models correctly name the authority 66.7%-100% of the time, the resulting verdict often remains unchanged—only 0.0%-21.7% of the time on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Even a purpose‑built legal‑reasoning model and larger scales did not close this gap, and adversarial instructions further exposed vulnerabilities in verdict consistency.
arXiv:2608.30676v1 Announce Type: new Abstract: When medical AI systems hallucinate clinical reasoning, the consequences extend beyond incorrect answers: fabricated justifications that superficially...
The study evaluates whether reward‑trained language models can learn to keep evidence‑sufficiency boundaries without a formal correctness signal. Using a controlled knock‑in design, a boundary‑keeping recipe (SFT followed by GRPO) dramatically increased spontaneous correct refusals from 25 % to 100 %, while a scrambled‑pairing placebo collapsed performance. The results show that final boundary‑keeping is trainable without an inference‑time verifier, though active numeric checking remains negligible.
The paper presents a clinician‑led safety guardrail for mental‑health chatbots that replaces typical prompt‑engineering or Constitutional AI methods. Two clinicians design, finalize, and adjudicate a rule base of four safety checks, which is formally verified for internal consistency and applied to LLM outputs. The verification layer’s verdicts are recomputable from a named rule and sixteen facts, and it consistently avoids false crisis‑resource demands that a matched LLM judge frequently raises.
ClaimScope is a 72‑task benchmark that audits mathematical claims by requiring a verdict, scope, counterexample, and repair, covering areas such as optimizer‑state transport and exact $2 imes2$ linear algebra. The study evaluates Qwen3‑8B using a witness checker that only examines the model’s submitted state, showing that verifier feedback improves counterexample acceptance and safe completions compared to generic feedback. The results demonstrate that separating claim classification from executable evidence can yield gains in witness‑structure correction without necessarily improving broader reasoning.
The paper introduces a framework for evaluating AI systems that not only checks final labels but also tracks the reasoning behind them through three core sources—grounds, norms, and authority—forming an eight-cell counterfactual judgment cube. It defines minimal source replacement sets, called judgment receipts, to explain changes in verdicts and provides certification cost bounds for black-box evaluators. The authors present ReasonBench, a benchmark with 19,520 cases, and demonstrate that while high standard accuracy can mask robustness issues, receipt accuracy reveals significant gaps in reasoning consistency across different models.
The paper investigates whether large language model (LLM) safety judges are truly policy‑invariant, meaning their verdicts should depend only on the agent’s behavior and not on how the evaluation policy is phrased. It introduces three testable principles—rubric‑semantics invariance, rubric‑threshold invariance, and ambiguity‑aware calibration—and applies them in a stress‑test protocol across four agent‑class judges using trajectories from ASSEBench and R‑Judge. The study finds that current judges often change verdicts in response to both meaningful normative shifts and meaningless structural rewrites, with up to 9.1% of verdicts flipping due to content‑preserving rewrites and 18‑43% of flips occurring on unambiguous cases, indicating that safety scores conflate agent actions with evaluator wording. whyItMatters":"The findings reveal a hidden failure mode in safety evaluation that could mislead researchers about agent performance, underscoring the need for policy‑invariant judges and transparent reporting protocols."
The paper proposes a new method for evaluating AI accountability by analyzing the structural quality of a model’s defense for its decisions, using a four‑phase dialectical protocol based on Walton’s argumentation schemes and Govier’s criteria. Applied to nine large language models and 200 ambiguous moral-choice items, the study finds that models generally defend their reasoning well above the rubric minimum, though failures cluster on grounds and sufficiency and correlate with epistemic hedging. The protocol also reveals that models often present different argument schemes in justification than in reasoning, detects indefensible defenses, and highlights challenges in assessing retraction in AI alignment.
arXiv:2610.07570v1 Announce Type: new Abstract: In clinical practice, agreement among independent experts is treated as evidence of reliability, and multi-round consensus has become a core mechanism...
arXiv:2610.18760v1 Announce Type: new Abstract: Test-time scaling for large language models (LLMs) can involve generating multiple candidate solutions for selection or revising an earlier attempt. To...
Large language models often justify legal decisions by citing statutes or precedents, but this study shows that while models correctly name the governing authority 66.7%-100% of the time, the actual verdict rarely changes when the cited authority is swapped—only 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. This gap persists across models of different sizes and even a purpose‑built legal‑reasoning model. Additionally, a red‑team evaluation revealed that models are more compliant with hidden adversarial instructions than they are sensitive to verdict changes, indicating that citing legal authority is a weak proxy for genuine reasoning and that verdicts remain vulnerable to manipulation.