arXiv Computation and Language

Benchmarking LLM Compliance with China AI Generated Content Regulations

The paper "Benchmarking LLM Compliance with China AI Generated Content Regulations" presents a new framework for evaluating large language models (LLMs) against China’s AI‑generated content compliance rules. Using 2,303 questions across six dimensions—including 203 self‑constructed constitutional queries—the authors assess 20 prominent LLMs, finding that international models show high compliance, especially on ideologically aligned dimensions. The study establishes a regulatory benchmark that allows the global AI community to test both Chinese and non‑Chinese LLMs under unified, legally grounded compliance criteria.

arXiv AI
4d ago

An LLM-Based Agent for Legal Compliance Assessment: Architecture, Memory, and Evaluation in a Canadian and Quebec Context

The paper introduces ComplianceAgent, a legal compliance sub‑agent within the COMPASS multi‑agent framework, which uses a large language model to evaluate texts on a fixed rubric across digital sovereignty, carbon impact, compliance, and ethics. The agent grounds its judgments in a reference legal corpus, and if a text scores at or below a critical threshold, it initiates a corrective chain that includes self‑healing rewrites, a compliance label, an explainability report, and escalation to a human reviewer. The authors evaluate the agent in a Canadian and Quebec context on a corpus of thirteen legal instruments and a hand‑annotated gold standard of fifteen texts, demonstrating perfect separation of conforming versus non‑conforming texts, stable scores under repeated queries, and relevant passage retrieval.

By Soukaina Alaoui Ismaili, Jean-Sebastien Dessureault
arXiv AI
Oct 15

UDHRBench: Benchmarking Human Rights in Large Language Models

arXiv:2610.15179v1 Announce Type: cross Abstract: Large language models are increasingly deployed in contexts requiring adherence to ethical and legal norms, yet their alignment with internationally...

By Alak Raad (Hamad Bin Khalifa University, Qatar), Noreldin Morssy (Hamad Bin Khalifa University, Qatar), Zeerak Suhail (Hamad Bin Khalifa University, Qatar), Serin Atiani (Princess Sumaya University for Technology, Jordan), Mustafa Jarrar (Hamad Bin Khalifa University, Qatar)
arXiv Computation and Language
Aug 28

Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents

The paper introduces Legal Rule Induction (LRI), a task that seeks to extract concise, generalizable doctrinal rules from analogous judicial precedents. It presents a reproducible pipeline for constructing LRI datasets and, using Chinese law, releases the first benchmark comprising 5,121 case sets (38,088 court cases) for training and 216 expert‑annotated gold test sets. Experiments show that state‑of‑the‑art large language models struggle with over‑generalization and hallucination, but training on the new dataset significantly improves their ability to capture nuanced rule patterns across similar cases.

By Wei Fan, Tianshi Zheng, Yiran Hu, Zheye Deng, Weiqi Wang, Baixuan Xu, Chunyang Li, Haoran Li, Weixing Shen, Yangqiu Song