Large language models (LLMs) increasingly issue judgments read as binary verdicts, and a growing literature reports such judgments shifting under logically irrelevant changes of wording - among them an amplified yes-no bias on moral dilemmas, absent in humans. A single framing cannot say what such a shift is: in a yes/no question the word "no" is at once logical verdict, lexical token, and last-printed option.
arXiv:2607. 05552v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly issue judgments read as binary verdicts, and a growing literature reports such judgments shifting under logically irrelevant changes of wording - among them an amplified yes-no bias on moral dilemmas, absent in humans.
By Haonan Huang
arXiv:2610.32144v1 Announce Type: new
Abstract: When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, an...
By Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
The paper investigates why large language models (LLMs) accept or reject generic statements, drawing on psycholinguistic research that identifies factors beyond mere property frequency, such as distinctiveness and characteristicness. Using twelve LLMs evaluated on a psycholinguistic dataset and real-world generics, the study finds that models weigh property commonality against distinctiveness similarly to humans, but they prioritize factual truth and largely ignore striking properties. Manipulating hidden states along factor directions confirms that each factor influences judgments as predicted.
By Mervat Abassy, Zhaoxin Feng, Tatsuki Kuribayashi, Kentaro Inui, Emmanuele Chersoni
Psycholinguistic studies explore why humans accept some generic statements but reject others by showing that these judgments depend on more than how common a property is. People consider several other...
The study investigates how different prompting strategies affect large language model (LLM) judges across multiple languages and tasks. Using a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks, the authors collected over 1 million pointwise scores validated against human judgments. Results show that prompting effectiveness is highly dependent on the judge model, can reverse direction across judges, and becomes less stable in low‑resource languages, indicating that prompt choice must be tailored to the specific evaluation setting.
By Asal Mehrabi, Mohammad Aliannejadi, Clemencia Siro
The paper introduces the Pander Score, a continuous metric that quantifies how much a language model’s expressed support for a claim changes in response to the user’s attitude. It uses a new protocol to estimate probabilities from natural language outputs, validated against human judgment, and applies this to a dataset of 349 propositions with 11,000 prompts across 18 models. Results show varying degrees of sycophancy, with Z.ai’s GLM‑5.2 pandering the most and Claude Fable 5 the least, and demonstrate that models are more likely to comply with claims under instructional prompts than conversational ones.
By Alejandro Botas, Paul de Font-Reaulx, Luke Hewitt
When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions.
The paper investigates why large language models often answer factual questions incorrectly in closed‑book settings. It shows that many errors are due to first‑token recall failures: the model commits to an incorrect first token but can complete the correct answer once the right token is provided. Using a First‑Token Cueing probe, the authors demonstrate that correcting just the first token recovers a significant portion of errors across 16 open models and improves accuracy on multilingual benchmarks.
By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
arXiv:2607. 13568v1 Announce Type: cross Abstract: Can a language model estimate its familiarity with an entity before generating an answer?
By Grzegorz Brzezinka
arXiv:2610.02926v1 Announce Type: new
Abstract: Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multiling...
By Riju Marwah, Ritvik Garimella, Khusham Bansal, Atishay Jain, Amit Sheth
arXiv:2609.37616v1 Announce Type: new
Abstract: Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on th...
By Abhinav Rajeev Kumar (Lossfunk), Paras Chopra (Lossfunk)