A benchmark score means nothing without knowing what a trivial method achieves and what the best possible method could achieve. We construct both bounds for a task with a rare kind of ground truth: predicting which sentences a crowd of readers -- highlighting for their own purposes, unpaid, uninstructed, and blind to each other -- marked in 120 web documents.
The study investigates how demographic identity is represented in a language model, using representational similarity analysis against Pew survey data across 169 demographic cells. It finds that standard last‑token read‑outs underestimate the model’s fidelity, while specific attention heads (notably L11 H16) capture demographic structure more accurately, though race‑based types remain weak. Causal interventions reveal that high fidelity does not guarantee causal use, and a 128‑dimensional probe of a single head improves alignment with survey truth but fails to recover per‑question group ordering.
arXiv:2608.18768v2 Announce Type: replace
Abstract: Large language models are widely used to simulate survey respondents, yet their outputs are homogeneous and unfaithful to real inter-group differen...
By Fathin Difa Robbani
arXiv:2608. 15980v1 Announce Type: cross Abstract: Preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail.
By Anik Jha
The study introduces PV‑SST, a peer‑voted social‑platform testbed, and conducts a preregistered matched‑exposure experiment across four topics, four seeds, four model families, and three larger variants, totaling 448 trials. Results show that feeding agents a ranked list of prior peer posts increases lexical similarity in both the core panel and larger variants, but does not produce a reliable advantage in opinion alignment or survival rates. The only robust finding is lexical convergence driven by the peer‑ranked feed, with no consistent coordination benefit across models or topics.
By Rana Muhammad Usman, Dominic Williamson
The paper investigates how sentence‑specificity scores can guide the selection of revisions in collaborative technical documentation. It compares two predictors—SpeciTeller and a target‑adapted model by Ko et al.—across Wikipedia and three technical corpora, finding that the predictors rank sentences differently and that SpeciTeller can improve direction‑valid selection rates in certain datasets. The study also shows that filtering and token‑length adjustments alter but do not reconcile these ranking differences.
By Rocker D'Antonio, Thomas Benton Townsend, Dimitrios Michael Manias