arXiv AI October 31, 2026 By Robert Pasternak

Prompts and probabilities in the linguistic assessment of language models: Evidence from semantic focus-sensitivity

The study examines how language models (LMs) perform on semantic focus-sensitivity tasks using two evaluation methods: explicit prompting and token-probability-based scoring. Across models of varying sizes, token probabilities consistently achieve above-chance performance, while prompting generally aligns with prior findings but can surpass token probabilities in larger models. The authors attribute this to LMs’ limited use of focus markers, which can distort token-probability results, and demonstrate that prompting remains robust even when focus markers are randomized.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv Computation and Language

Oct 27

Large Language Model probabilities cannot reliably distinguish between possible and impossible language

arXiv:2509.15114v2 Announce Type: replace Abstract: A controversial test for Large Language Models concerns the ability to discern possible from impossible language. While some evidence attests to th...

By Evelina Leivada, Raquel Montero, Paolo Morosi, Natalia Moskvina, Tamara Serrano, Nikoleta Pantelidou, Marcel Aguilar, Fritz Guenther

llms benchmarks

arXiv AI

3d ago

Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing

The paper introduces the "Prompt Lottery" phenomenon, where paraphrastic rephrasings of judge prompts in LLM-as-a-Judge (LLMaJ) lead to inconsistent judgments and can even reverse model rankings. A large-scale study across various tasks, judge families, and model backbones demonstrates that these prompt variations significantly alter evaluation outcomes. A user study shows that LLMaJ judges are 6–9 times more sensitive to prompt rephrasings than humans, and that aggregating multiple prompts improves robustness by focusing on consistent judgments.

By Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen

llms

arXiv Computation and Language

Oct 7

Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing

arXiv:2610.07936v1 Announce Type: new Abstract: Systematicity, the probabilistic mapping of form to meaning, permeates language at all levels, and sublexical cues have been shown to govern human pseu...

By Jing Chen, Giulia Loca, Simona Amenta, Marco Marelli

llms nlp safety

arXiv Machine Learning

Jun 5

CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction

arXiv:2606. 05799v1 Announce Type: new Abstract: Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's {\em behavioral robustness} to irrelevant or misleading information.

By Mohammad Anas Jawad, Cornelia Caragea

llms benchmarks

arXiv AI

Oct 27

Prompt PerturbAItions: Mechanisms of LLM Response Inconsistencies

arXiv:2605.30913v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly deployed in conversational settings where user phrasing ranges from polite to casual to toxic,...

By Soorya Ram Shimgekar, Agam Goyal, Amruta Parulekar, Joshua Chen, Yian Wang, Navin Kumar, Hari Sundaram, Eshwar Chandrasekharan, Koustuv Saha

llms benchmarks safety

Hugging Face Trending Papers

Jun 23

Sentence-Level Contextual Entrainment in Large Language Models

Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens.