arXiv AI
ConsistencyAI: A Benchmark to Assess LLMs' Factual Consistency When Responding to Different Demographic Groups
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 19
Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference
The paper introduces the Pander Score, a continuous metric that quantifies how much a language model’s expressed support for a claim changes in response to the user’s attitude. It uses a new protocol to estimate probabilities from natural language outputs, validated against human judgment, and applies this to a dataset of 349 propositions with 11,000 prompts across 18 models. Results show varying degrees of sycophancy, with Z.ai’s GLM‑5.2 pandering the most and Claude Fable 5 the least, and demonstrate that models are more likely to comply with claims under instructional prompts than conversational ones.
arXiv AI
Jun 9
The AI Epistemic Deference Index: A Continuous Measure of Sycophancy
arXiv:2606. 07897v1 Announce Type: new Abstract: Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user.
arXiv AI
Jul 22
Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs
arXiv:2607. 19243v1 Announce Type: cross Abstract: Although Large Language Models (LLMs) demonstrate remarkable multilingual fluency, their internal knowledge representations remain disproportionately biased toward high-resource languages.
arXiv Computation and Language
Sep 3
When Persona Attributes Improve Population Alignment in Large Language Models
The paper investigates how persona prompting—using short textual descriptions of individuals—to align large language models (LLMs) with human survey responses. It examines the impact of selecting different persona attributes and finds that not all attribute combinations improve performance, suggesting that the variation in human responses to survey questions may explain mixed results. The study evaluates multiple attribute selection methods across four social surveys, two countries, six LLMs, and twenty prediction tasks, offering guidance on when persona prompting is beneficial and which attribute choices are most effective.
arXiv Computation and Language
2d ago
ActTraitBench: Quantifying the Knowledge-Decision Gap in Large Language Models via Human-Grounded Behavioral Validation
arXiv:2605.29791v2 Announce Type: replace Abstract: While Large Language Models (LLMs) can convincingly simulate personas in explicit self-reports, they often deviate in implicit behavioral decisions...
arXiv Computation and Language
2d ago
Do LLMs Make More Mistakes If They Do Not Believe the Input Data?
The study investigates whether large language models (LLMs) are more prone to errors when they doubt the plausibility of input data, a phenomenon termed context‑memory conflict. Using non‑English and low‑resource language datasets, the authors generate text from factual, counterfactual, and fictional RDF triples in English, Czech, Slovak, and Upper Sorbian, and evaluate faithfulness with both human annotations and an LLM judge (Kimi K3). Contrary to expectations, the results show only a weak context‑memory conflict: counterfactual inputs receive slightly lower faithfulness scores than factual ones, and the choice of LLM judge can significantly affect perceived conflict strength.