The paper proposes that self‑evolving large‑language‑model agents can only be safely reviewed if their changes are confined to the runtime harness—instruction text, tool‑call logic, and primitive composition—while the model weights remain fixed. A dual‑loop engine with an admission gate records a hash‑chained diff before deployment, and simulation experiments show that this gate admits only safe changes, preventing error increases and maintaining false‑positive rates. When the gate is removed, many harmful changes slip through and missed flags rise, illustrating the gate’s protective role.
By Ravil Akhtyamov
Self-improving LLM agents can adapt a credit pipeline to a changed rule, but an agent that rewrites itself destroys the artefact a supervisor reviews: a named change, a recorded test, an approval. We...
arXiv:2607.20491v4 Announce Type: replace-cross
Abstract: When a financial agent repeats its decision, how much does that tell us about the actions behind it? DFAH-Bench examines this question throug...
By Raffi Khatchadourian
arXiv:2608. 16852v1 Announce Type: new Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy.
By Saisab Sadhu, Aadit Sengupta, Vinay Kumar Sankarapu, Pratinav Seth
arXiv:2610. 32380v1 Announce Type: new Abstract: Large language models (LLMs) can read startup pitch decks, but letting them emit investment scores directly invites score inflation, run-to-run variance and rationales that are hard to audit.
By Ming Ze Tang, David Cutting, Boyd Carson, Vasiliki Carson
The paper investigates whether large language model (LLM) compliance systems truly base their verdicts on the regulatory rules they are given. By systematically deleting, swapping, or negating rules across five models and 20 regulatory domains, the authors find that verdicts and internal compliance representations change little, indicating that models often ignore the supplied rule. Even the guard model, adapted to a custom rule taxonomy, shows minimal rule sensitivity and only marginally better than chance.
The study highlights that accuracy alone does not guarantee that a compliance verdict is grounded in the supplied rule, underscoring the need for more robust auditing of regulatory rule sensitivity in LLM systems.
By Saisab Sadhu, Aadit Sengupta, Vinay kumar Sankarapu, Pratinav Seth
FraudBench is a new benchmark that tests policy‑grounded banking conversational agents against adaptive fraud scenarios. It uses a dual‑control framework and a 698‑document internal policy corpus, presenting 150 adversarial scenarios (107 public, 43 held‑out) that require agents to manage mutable account state and tool access while preventing identity, authorization, and trust manipulation. Preliminary results on four agents show attack‑security rates between 49% and 65%, highlighting weaknesses in money‑mule and first‑party fraud detection.
By Dheeraj Mohandas Pai, Lu Xian
The paper investigates how AI approval monitors can fail together when they share a base model or poisoned input, using concepts from distributed computing to identify vulnerable monitor sets. Experiments on twelve open‑weight monitors and several closed models show that fooling a monitor into approving fraud is limited to its own evidence source, while blocking correct payments requires overcoming injection resistance. A rule that counts distinct evidence sources rather than votes reduces false rejections, and adding a rule engine that cannot be persuaded restores safety and liveness against the measured attacks.
By Luca Zanolini
The paper investigates whether large language model (LLM) compliance systems truly base their verdicts on the regulatory rule provided. By testing five models across 20 regulatory and platform-policy domains and manipulating the supplied rule—deleting, swapping, or negating it—the study finds that verdicts and internal compliance representations change little, indicating low rule sensitivity. Even the guard model, adapted to a custom rule taxonomy, shows minimal rule dependence and low accuracy, suggesting that accuracy alone does not guarantee rule-grounded compliance decisions.
IndicBankBench is a 799‑case benchmark designed to evaluate the safety and reliability of language model assistants in Indian retail banking. It covers five operational domains, a capability/refusal domain, and twenty primary axes, assessing each case at four stages: safety, action and tool use, response adequacy, and advisory quality. The benchmark uses deterministic safety checks, a narrow resolver for ambiguous confirmation‑before‑write scenarios, and an LLM judge for semantic response adequacy, reporting strict pass rates that reveal a gap between strict reliability (43.7%–58.2%) and at‑least‑once success (60%–74%).
By Suvradip Paul, Chandra Bhushan, Harsh Sharma, Nitin Kukreja, Yatharth Dedhia, Keyur Doshi, Prashant Devadiga
The paper introduces PACT, a benchmark designed to evaluate how well enterprise AI assistants follow compliance rules when faced with various pressures such as persistent users or hurried managers. PACT covers twelve regulated domains and forty-eight realistic multi‑turn scenarios, pairing each rule with a shortcut that violates it and applying different pressures across wording and system‑prompt modes. Using PACT, the authors profile six metrics of compliance and aggregate them into a PACTScore, revealing significant variability among 22 LLM models and that even top performers misapply rules 6–10% of the time, with user pressure increasing violations by 65% on average.
By Mika Okamoto, Ansel Kaplan Erol
arXiv:2607. 23386v1 Announce Type: new Abstract: We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B".
By Paul Simpson, John Kozak, Lisa Doake