arXiv AI
Aug 12
Toward Human Rights Benchmarking for LLMs: A Pilot Methodology
arXiv:2608. 10268v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how.
UDHRBench: Benchmarking Human Rights in Large Language Models
Related stories
arXiv AI
Sep 23
IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law
arXiv:2609.22529v1 Announce Type: new Abstract: International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its t...
arXiv AI
Aug 14
Agreement Is Not Alignment: Divergent Moral Grounds in Human and LLM Ethical Judgments
arXiv:2608. 12368v1 Announce Type: new Abstract: Agreement with human judgments is a common proxy for evaluating the alignment of large language models (LLMs).
arXiv Computation and Language
Sep 1
LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks
arXiv:2606.09389v2 Announce Type: replace Abstract: As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses...
arXiv Computation and Language
Sep 25
Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms
The paper investigates whether large language models (LLMs) assess politeness in ways that match human judgments. Using two English datasets—one with continuous ratings and another with three‑way categorical labels—the authors compare seven LLMs to human annotations. They find that models agree more with each other than with humans, show systematic neutral bias in categorical predictions, and that alignment varies with explicit linguistic cues and rapport‑building strategies.
arXiv Computation and Language
3d ago
TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics
arXiv:2610.29477v1 Announce Type: new Abstract: Codes of professional ethics guide and restrict specialists, who increasingly work with assistants built on large language models. Such an assistant ca...
arXiv AI
Jul 1
Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
arXiv:2606. 23375v2 Announce Type: replace-cross Abstract: While the wider applicability of LLMs in the legal field is currently debated due to their reliability and the gravity of any errors, narrow uses with well-understood and mitigated risks have emerged.
arXiv AI
Jun 2
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
arXiv:2605. 28183v2 Announce Type: replace-cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law.
Hugging Face Trending Papers
Sep 24
Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms
The paper investigates whether large language models (LLMs) assess politeness in ways that match human judgments. Using two English datasets—one with continuous ratings and another with three‑way categorical labels—the authors find that LLMs agree more with each other than with humans. They observe that model–human alignment depends on explicit linguistic cues, while misaligned cases often involve rapport‑building strategies. Additionally, models tend to overproduce Neutral labels and underpredict Impolite labels, a pattern that persists even when expert consensus is used as a reference.
arXiv AI
Aug 7
Poli-Bias: Understanding and Measuring Large Language Model Biases in International Political Conflicts
arXiv:2608. 06123v1 Announce Type: new Abstract: Measuring political bias in large language models (LLMs) remains challenging as it can manifest through subtle differences in framing, argumentation, and legal reasoning that are difficult to capture with a single metric.
arXiv AI
Aug 10
Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
arXiv:2605. 29738v2 Announce Type: replace-cross Abstract: Legal NLP benchmarks overwhelmingly evaluate a single language or aggregate tasks that differ fundamentally across jurisdictions, making cross-lingual comparison impossible.
arXiv AI
Sep 1
Statutory AI: Aligning Large Language Models With Legal Norms
arXiv:2608.28593v1 Announce Type: new Abstract: With the increasing development of AI regulatory frameworks, ensuring that artificial intelligence systems, particularly generative models, operate in...