← Back to all news
arXiv Computation and Language By Yuqi Wu, Deyang Li, Katharine Hibbard, Ron Oswald, David Sinclair, Adegboyega Sapara, Kalutota Kuruwe Arachchige Sajana Indula Samarasinghe, Abhinav Pillai, Reham Shalaby, Riffat Alam, Xin-Min Li, Yanbo Zhang, Jie Chen

The Schema Coherence Inventory: Measuring Whether an Assigned Belief Organizes a Language Model's Cognition

Read the original on arXiv Computation and Language →

The paper introduces the Schema Coherence Inventory (SCI), a 96-item forced-choice test designed to determine whether a belief assigned to a persona-conditioned language model actually organizes its behavior or merely reflects affect. SCI evaluates belief organization across eight levels of Beck's cognitive model, using length‑normalized log‑likelihood scores and a target‑specificity criterion to distinguish true belief influence from mood effects. Validation across five model backbones (1.7B–14B parameters) and nine clinicians shows high reliability and discriminant validity, and a 48‑item short form correlates strongly with the full inventory.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Aug 10

Natural Language Processing Psychometrics

arXiv:2608. 07316v1 Announce Type: cross Abstract: Natural Language Processing (NLP) models predicting mental health outcomes rarely specify what they measure: contextual knowledge, emotional content, or syntactic structure.

By Edoardo Sebastiano De Duro, Emma Franchino, Massimo Stella
llmsnlp
More like this →
arXiv AI
Sep 1

The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometrics

The study investigates how different prompt components affect language model responses in psychometric tests. By crossing five distinct baseline personas with five variants of each prompt element—persona wording, task instruction, item wording, and option symbol—the authors measure response shifts using the 1‑Wasserstein distance. Their analysis of 13 small open‑weight language models on the Big Five Inventory and Short Dark Triad reveals that task instruction and option symbol changes often cause more variation than paraphrasing the persona or item, with prompt artifacts explaining over 50% of the variation for many items.

By Nils Schwager, Christoph Hau, Simon M\"unker, Achim Rettinger
llms
More like this →
arXiv Computation and Language
3d ago

Which Answer Did the Seventeen-Year-Old Write? A Behavioral Case Study of Author-Age Misattribution, Miscalibrated Confidence, and Non-Corrective Self-Explanation in Six Frontier Language Models

The paper reports a structured case study in which six frontier language models were asked to match handwritten exam answers to the ages of the writers. Three of the models performed at chance on the matching task while still reporting high confidence, and the youngest writer’s answer was misattributed to an adult in up to 90% of trials. Removing the multiple‑choice scaffold and changing institutional framing further degraded accuracy, while a single audience‑awareness instruction improved performance for one model but had no effect on the weakest models.

By Brianne Lee
llmssafety
More like this →