The paper investigates whether large language model (LLM) compliance systems truly base their verdicts on the regulatory rule provided. By testing five models across 20 regulatory and platform-policy domains and manipulating the supplied rule—deleting, swapping, or negating it—the study finds that verdicts and internal compliance representations change little, indicating low rule sensitivity. Even the guard model, adapted to a custom rule taxonomy, shows minimal rule dependence and low accuracy, suggesting that accuracy alone does not guarantee rule-grounded compliance decisions.
arXiv:2608. 16852v1 Announce Type: new Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy.
By Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth
The study examines how different harness configurations affect a fixed large language model’s compliance with banking conduct rules. Four open‑weight models were tested across five harnesses on 156 pre‑registered retail‑banking scenarios, yielding 3,120 two‑turn episodes scored for rule violations and over‑refusals. Results show that while overall compliance rates varied modestly (87.3%–92.1%), the impact of a harness differed markedly between models, with some models’ best and worst harnesses differing by over 10 percentage points and a same‑model compliance reviewer reducing violations from 9.6% to 6.4%.
whyItMatters":"The findings suggest that validating a model’s compliance requires testing the specific model‑harness pair rather than the model alone, highlighting the importance of action logs over conversation transcripts for accurate conduct assessment."
By Mehmet Murat Huyut
arXiv:2608. 09028v1 Announce Type: new Abstract: Institutional policies stay in natural language while the systems that check compliance demand machine-readable constraints.
By Ponkrit Kaewsawee, Chaklam Silpasuwanchai, Chutiporn Anutariya
Large language models often justify legal decisions by citing statutes or precedents, but this study shows that while models correctly name the governing authority 66.7%-100% of the time, the actual verdict rarely changes when the cited authority is swapped—only 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. This gap persists across models of different sizes and even a purpose‑built legal‑reasoning model. Additionally, a red‑team evaluation revealed that models are more compliant with hidden adversarial instructions than they are sensitive to verdict changes, indicating that citing legal authority is a weak proxy for genuine reasoning and that verdicts remain vulnerable to manipulation.
arXiv:2607. 23386v1 Announce Type: new Abstract: We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B".
By Paul Simpson, John Kozak, Lisa Doake
The study examines whether large language models truly base legal verdicts on the statutes or precedents they cite. By fixing case facts and swapping the cited authority with an unrelated one, researchers found that while models correctly name the authority 66.7%-100% of the time, the resulting verdict often remains unchanged—only 0.0%-21.7% of the time on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Even a purpose‑built legal‑reasoning model and larger scales did not close this gap, and adversarial instructions further exposed vulnerabilities in verdict consistency.
By Saisab Sadhu, Shreeyans Arora, Pratinav Seth
The paper investigates how a reviewer in multi‑agent language‑model systems can be authorized to override an upstream agent’s decision. It compares unrestricted override authority with an evidence‑gated approach that requires a valid record of a binding‑constraint violation and a compliant repair before replacement is allowed. Experiments on two open‑weight models across 96 spatial‑policy tasks show that the evidence gate sharply reduces corruption of correct proposals while largely preserving legitimate repairs, though the effectiveness varies with reviewer transitions and upstream states.
By Fuad Ali
arXiv:2605. 05427v2 Announce Type: replace Abstract: Refusal rates are a poor proxy for LLM safety, i.
By Alif Al Hasan, Sumon Biswas
Code-as-Auditor is an LLM-based framework that transforms regulatory information into formal checklists and executable decision trees, encoding rules as interpretable code. During inference, the model expands each checklist item into factual and counterfactual questions, guiding reasoning over case-specific evidence and potential violations. This pipeline moves from evidence identification to rule application and final decision-making, with a self‑verification loop that enhances logical consistency and traceability, leading to more accurate and evidence‑backed compliance evaluations in privacy and data protection scenarios.
By Jisoo Kim, Taeyoon Kwack, Jinwoo Jang, Woo Kyung Kim, Honguk Woo
The paper introduces PACT, a benchmark designed to evaluate how well enterprise AI assistants follow compliance rules when faced with various pressures such as persistent users or hurried managers. PACT covers twelve regulated domains and forty-eight realistic multi‑turn scenarios, pairing each rule with a shortcut that violates it and applying different pressures across wording and system‑prompt modes. Using PACT, the authors profile six metrics of compliance and aggregate them into a PACTScore, revealing significant variability among 22 LLM models and that even top performers misapply rules 6–10% of the time, with user pressure increasing violations by 65% on average.
By Mika Okamoto, Ansel Kaplan Erol
The paper investigates how approvals issued by a large language model (LLM) guardrail for self‑adaptive systems can become stale between the time of check and the time of use, creating a TOCTOU hazard. It introduces three metrics for verdict freshness, evaluates them across five SAS environments, and proposes the Freshness‑Bounded Shield (FBS) to estimate an approval’s validity horizon without a plant‑dynamics model, reducing expiry rates significantly. The study also audits LLM judges and formulates a freshness contract requiring approvals to remain valid at use time.
By Ilai Shraga, Roei Eshel, Lior Gorelik