arXiv AI By Mario Vega-Barbas, Lidia Mora-Valenciano, Iv\'an Pau, Fernando Seoane, Farhad Abtahi

Sixteen models, fewer than two voices: measuring ensemble dispersion where no answer is uniquely correct

Read the original on arXiv AI →

The study evaluates how sixteen language models from ten families generate diverse formulations of a psychotherapeutic case, finding an average semantic diversity of 1.69 distinct formulations versus 1.43 for a single-model baseline. It introduces the Vendi Score to quantify diversity and defines a per-model dissent metric to identify the most divergent voice within an ensemble. The analysis shows that model identity significantly influences dissent, but this effect varies across model pairs and panel compositions, indicating that ensemble dispersion is a measurable property rather than an assumed one.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
Sep 21

How Many Humans Is a Judge Panel Worth?

arXiv:2609.21277v1 Announce Type: cross Abstract: How many human judgments does a panel of language models represent? The answer depends on what is matched. We audit categorical judge panels against...

By Chao Li, Yingying Yu, Yunfeng Li
Hugging Face Trending Papers
Aug 19

Readable, Faithful, Used: Three Dissociable Properties of Demographic Identity in a Language Model

The study investigates how demographic identity is represented in a language model, using representational similarity analysis against Pew survey data across 169 demographic cells. It finds that standard last‑token read‑outs underestimate the model’s fidelity, while specific attention heads (notably L11 H16) capture demographic structure more accurately, though race‑based types remain weak. Causal interventions reveal that high fidelity does not guarantee causal use, and a 128‑dimensional probe of a single head improves alignment with survey truth but fails to recover per‑question group ordering.