arXiv AI
Don't Ask: Understanding What Numeric Self-Reports Measure in Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 14
Self-Referential Induction Increases Response Instability Relative to Unresolvable and Verifiable Questions in Large Language Models
arXiv:2608. 13258v1 Announce Type: cross Abstract: Self-referential prompting has been shown to reliably induce large language models to produce first-person reports resembling subjective experience, but no prior work measures how consistent these reports are across repeated, independent trials, or how that consistency compares to the model's behavior on other kinds of open-ended questions.
arXiv AI
3d ago
Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring
The study investigates how different rating scale designs affect scores produced by large language models (LLMs) in text scoring tasks. By varying format, range, and sign across tasks like political leaning, sentiment, and toxicity, the authors find that a single design change can shift normalized scores by about 0.10 and alter the category of roughly 20% of texts. The research highlights that rubric-based prompting does not mitigate this sensitivity and emphasizes the need for robustness checks when using LLMs for scalar scoring.
arXiv AI
Oct 7
Knowing When Not to Answer: Cross-Domain and Multi-Turn Generalization of Latent Underspecification Signals
The paper presents a new multi‑turn benchmark of 423 conversations with 1,661 labeled turn‑states to study when language models should refrain from answering. It shows that probes for unanswerability transfer well across datasets that share the same underlying signal, but fail to generalise to other forms of epistemic uncertainty. While a calibrated probe can identify underspecified turns more accurately than chance, it does not consistently improve overall generation quality compared to standard methods.
arXiv Machine Learning
Aug 14
A Probe Direction Is a Property of Its Prompt
arXiv:2608. 13329v1 Announce Type: new Abstract: A model that behaves differently when it senses it is being tested would undermine the evaluations we rely on, so recent work has sought to read that sense directly from a model's activations.
arXiv AI
Jun 19
Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
arXiv:2606. 20205v1 Announce Type: new Abstract: Psychological instruments designed for humans are increasingly used to assign large language models (LLMs) stable psychological profiles that affect their usability, safety assessment, and use as proxies for human participants in research.
arXiv AI
Jul 17
Introspection Fine-Tuning (IFT): Training Small LLMs to Introspect
arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?