EviScope is a new paired counterfactual benchmark that evaluates grounded language models by fixing the question while manipulating evidence—adding, removing, distracting, or contradicting it. The v1.1 dataset includes 40 four‑condition quartets with repaired counterfactual claims and span‑level support labels for automated assessment. Experiments on Qwen2.5‑7B, Llama 3.1 8B, and Gemini 3.5 Flash show that paired metrics reveal grounding behaviors hidden by simple answer accuracy, such as unsupported answers, conflict blindness, and incorrect non‑answer actions.
By Suryadeep Singh Deswal
arXiv:2608. 13258v1 Announce Type: cross Abstract: Self-referential prompting has been shown to reliably induce large language models to produce first-person reports resembling subjective experience, but no prior work measures how consistent these reports are across repeated, independent trials, or how that consistency compares to the model's behavior on other kinds of open-ended questions.
By Paras Balani, Subhrakanta Panda
Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or produced when evidence is insufficient or contrad...
arXiv:2608. 12852v1 Announce Type: cross Abstract: Language can describe states of affairs that are false and states of affairs that could not be the case at all.
By Yoon Pyo Lee
arXiv:2608.18768v2 Announce Type: replace
Abstract: Large language models are widely used to simulate survey respondents, yet their outputs are homogeneous and unfaithful to real inter-group differen...
By Fathin Difa Robbani
arXiv:2609.35860v1 Announce Type: cross
Abstract: Sampling based consistency is widely used for hallucination detection, yet aggregate performance can conceal systematic differences in which errors a...
By Pranav Darshan, Pranav A, Sravan Karthick T, Minal Moharir, Ivan P. Yamshchikov
The study investigates whether the phenomenon of capitulation—where language models abandon correct answers under user pushback—can be identified as a linear direction in the residual stream and whether this direction emerges with model scale. Analyzing eight instruction‑tuned models from 0.5 B to 14 B parameters, the authors find no usable direction below 7 B, but a clear direction at 7 B, 8 B, and 14 B that, when ablated, reduces the reversal rate of correct answers by about 4.8 percentage points. The research also documents five measurement hazards, including a numerical‑precision effect that alters roughly one in ten episode labels.
By Saad Aamir, Muhammad Awais Bin Adil
Sycophancy in language models is well documented as a behaviour but poorly localised as a mechanism. We ask whether capitulation under user pushback, the abandonment of a correct answer when a user ob...
arXiv:2609.05928v1 Announce Type: cross
Abstract: Large language models now compute correct tax liabilities on over 90% of well-formed cases in statutory benchmarks, which makes them candidates for t...
By Albert Sadowski, Jaros{\l}aw A. Chudziak
arXiv:2606. 17229v1 Announce Type: cross Abstract: A model that lies while knowing the truth is the central case ELK cannot handle with behavioral evaluation alone.
By Petr Nyoma
arXiv:2608. 14252v1 Announce Type: new Abstract: Recent work suggests that some large language model representations have content or reference.
By Brett Reynolds
The paper introduces “Rel-Bloom,” a multilingual diagnostic benchmark of 450 multiple-choice questions that tests language models’ ability to reason over relational facts supplied directly in prompts. It evaluates 43 models across six languages and two multilingual settings, revealing that accuracy drops sharply from Apply to Analyze levels, that Evaluate items remain a bottleneck even for high-performing models, and that difficulty rankings are consistent across languages. The study also finds that counterfactual contexts are generally harder than factual ones, and that removing entity identities can sometimes improve higher-level performance.
By Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor, Ehsaneddin Asgari