arXiv Computation and Language

Dual-Axis Policy Entailment: A Benchmark for Policy-Change Reasoning

Dual-Axis Policy Entailment (DAPE) is a benchmark that evaluates ordered pairs of policy directives on two dimensions: whether they govern the same policy domain and whether the later directive continues, broadens, narrows, reverses, or shifts to a different domain. The benchmark includes 1,000 expert-annotated pairs from Chinese government policy and a calibration protocol for annotation consistency. Models tested on DAPE perform poorly zero‑shot, but a fine‑tuned Qwen3‑14B model reaches the human ceiling in agreement.

arXiv Computation and Language
Sep 7

Can Large Language Models Anticipate Behavioral Responses to Social Policies? A Case of Pension Enrollment Prediction among China's Flexible Workers

The paper introduces FlexPension-LLM, a domain‑specialized large language model designed to predict pension enrollment among China’s flexible workers. By injecting policy‑grounded cues and using LoRA/SFT for rationale‑augmented supervision, the model achieves a Composite F1 score of 0.9316 on a blind split, outperforming several baselines and matching top commercial models. External survey tests confirm its robustness, with the narrowest performance range among strong systems.

By Yumiao Li, Peixin Liu, Donglin Di, Chen Li, Runhuan Feng
arXiv Computation and Language
Oct 9

The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection

The paper introduces MARS-Gov, a standpoint-aware multi-agent system designed to detect and mitigate bias in Dutch government documents. It combines legal evidence retrieval, open-set target screening, specialized jurors, conservative routing, and rewrite verification, and adds a dynamic "10th juror" when new bias groups are identified. On the DGDB benchmark, MARS-Gov achieves state‑of‑the‑art performance with an F1 score of 0.880, surpassing zero‑shot LLM detectors and supervised Dutch encoders while reducing unnecessary interventions to 2.5%.

By Yuchen Miao, Zijun Wang, Chang Han, Yurui Shi, Mingtai Zhang, Siyang Xu
arXiv AI
Aug 26

Comparing Explanations is Not Enough, Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models

arXiv:2602. 02304v3 Announce Type: replace Abstract: Large-scale foundation models exhibit behavioral shifts when subjected to interventions such as scaling, fine-tuning, reinforcement learning with human feedback, or in-context learning.

By Martino Ciaperoni, Marzio Di Vece, Roberto Pellungrini, Luca Pappalardo, Fosca Giannotti, Francesco Giannini
arXiv AI
6d ago

The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel

The paper introduces a method for measuring political positions in data‑sparse regions by treating a large language model as a single fallible rater within a panel, analogous to expert surveys. It demonstrates that this panel approach yields reliable agreement across multiple models and reveals that disagreements often stem from differing interpretations of the regime rather than factual errors. The authors provide a full rubric, prompt, and data set, using the Middle East and North Africa as a worked example.

By Tarek Gara
arXiv Computation and Language
2d ago

Precedent Labels and Facts Shift LLM Approval Decisions in Opposite Directions

The study investigates how language models (LLMs) handle refund or claim requests under written policies when presented with past cases that include both facts and recorded decisions. By separating these two components, the authors found that changing the recorded decisions of near‑copy past cases can increase approval rates by 30–55 percentage points, while altering the facts to support approval actually decreases approval rates, suggesting that the request appears weaker when compared to such facts. These effects were consistent across multiple models and both study‑specific and publicly paraphrased policies.

By Junwei Yao, Zike Yang, Yongxi Zhou, Yanxian Chen, Jianan Liu, Jiaxi Wen, Zhijun Wang
arXiv AI
Sep 7

Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny?

The paper proposes a new method for evaluating AI accountability by analyzing the structural quality of a model’s defense for its decisions, using a four‑phase dialectical protocol based on Walton’s argumentation schemes and Govier’s criteria. Applied to nine large language models and 200 ambiguous moral-choice items, the study finds that models generally defend their reasoning well above the rubric minimum, though failures cluster on grounds and sufficiency and correlate with epistemic hedging. The protocol also reveals that models often present different argument schemes in justification than in reasoning, detects indefensible defenses, and highlights challenges in assessing retraction in AI alignment.

By Daan R. Henselmans, Derck W. E. Prinzhorn, Arno Libert