arXiv AI

The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel

The paper introduces a method for measuring political positions in data‑sparse regions by treating a large language model as a single fallible rater within a panel, analogous to expert surveys. It demonstrates that this panel approach yields reliable agreement across multiple models and reveals that disagreements often stem from differing interpretations of the regime rather than factual errors. The authors provide a full rubric, prompt, and data set, using the Middle East and North Africa as a worked example.

arXiv Computation and Language
2d ago

How Green Is Your Manifesto? Scoring Environmental Positions in the 2024 EU Elections with an LLM Panel

The study introduces a panel of three large language models that score each sentence of political manifestos on environmental protection and climate change, producing averaged manifesto-level ratings. Applied to 41 2024 European Parliament election manifestos, the model’s rankings align closely with the Chapel Hill Expert Survey but tend to rate parties as greener than experts, suggesting a gap between promises and practice. The approach offers a low‑cost, multilingual, and reproducible alternative to traditional expert surveys and hand‑coding, with inter‑model disagreement serving as a built‑in reliability check.

By Franco Corti, Sajid Siraj
Hugging Face Trending Papers
Jul 9

Validity of LLMs as data annotators: AMALIA on authority

A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size.

Hugging Face Trending Papers
Jul 6

The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

Large language models (LLMs) increasingly issue judgments read as binary verdicts, and a growing literature reports such judgments shifting under logically irrelevant changes of wording - among them an amplified yes-no bias on moral dilemmas, absent in humans. A single framing cannot say what such a shift is: in a yes/no question the word "no" is at once logical verdict, lexical token, and last-printed option.

arXiv Computation and Language
2d ago

Dual-Axis Policy Entailment: A Benchmark for Policy-Change Reasoning

Dual-Axis Policy Entailment (DAPE) is a benchmark that evaluates ordered pairs of policy directives on two dimensions: whether they govern the same policy domain and whether the later directive continues, broadens, narrows, reverses, or shifts to a different domain. The benchmark includes 1,000 expert-annotated pairs from Chinese government policy and a calibration protocol for annotation consistency. Models tested on DAPE perform poorly zero‑shot, but a fine‑tuned Qwen3‑14B model reaches the human ceiling in agreement.

By Charles Chang (Duke Kunshan University), Yuen Yuen Ang (Johns Hopkins University), Bolun Sun (Northwestern University)