arXiv:2606. 08129v1 Announce Type: new Abstract: Large language models (LLMs) differ in architecture, training data, and optimization procedures, yet they may still develop similar internal inference patterns.
The paper investigates how adjectival modifiers affect the semantic plausibility of events, using the Adept benchmark of 16,000 English sentence pairs that differ by a single adjective. Experiments show that sentence transformers, despite being conceptually suited to the task, underperform compared to models like RoBERTa. The authors provide an error analysis and discuss the implications of their findings for future work on balancing training and test data.
arXiv:2606. 05486v1 Announce Type: cross Abstract: Prompt ambiguity is a common source of failure in large language models, but is difficult to localize because it is a latent property of the prompt, while existing attribution methods are designed to explain observable outputs such as logits or generated tokens.
The paper investigates whether adding Abstract Meaning Representation (AMR) data to large language models (LLMs) improves performance on downstream tasks. By reproducing recent studies and applying a consistent hyperparameter protocol, the authors find that text-only baselines match or surpass AMR-augmented models. A perplexity-based probe shows that AMR does not provide LLMs with additional relational knowledge, suggesting no clear benefit from AMR augmentation.
arXiv:2506.19549v2 Announce Type: replace-cross Abstract: Identifying which tokens and activations truly influence model predictions is critical for both the efficiency and interpretability of large...
The paper introduces the "Prompt Lottery" phenomenon, where paraphrastic rephrasings of judge prompts in LLM-as-a-Judge (LLMaJ) lead to inconsistent judgments and can even reverse model rankings. A large-scale study across various tasks, judge families, and model backbones demonstrates that these prompt variations significantly alter evaluation outcomes. A user study shows that LLMaJ judges are 6–9 times more sensitive to prompt rephrasings than humans, and that aggregating multiple prompts improves robustness by focusing on consistent judgments.
The paper evaluates four metrics—CKA, ANC, GMM dominance per token, and ILO—used to measure cross‑lingual representation sharing in multilingual language models. Across 21 models ranging from 125 M to 14 B parameters, the metrics disagree, and the authors attribute this to anisotropy, where representations cluster in a narrow embedding cone. Only ILO shows a strong, robust correlation with cross‑lingual transfer performance (Spearman’s ρ = 0.90) after controlling for model size, family, and task variation, leading the authors to recommend ILO as the primary metric alongside anisotropy diagnostics.
The study examines how language models (LMs) perform on semantic focus-sensitivity tasks using two evaluation methods: explicit prompting and token-probability-based scoring. Across models of varying sizes, token probabilities consistently achieve above-chance performance, while prompting generally aligns with prior findings but can surpass token probabilities in larger models. The authors attribute this to LMs’ limited use of focus markers, which can distort token-probability results, and demonstrate that prompting remains robust even when focus markers are randomized.
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy.
arXiv:2603. 01437v2 Announce Type: replace Abstract: As chain of thought (CoT) has become central to scaling reasoning capabilities in large language models (LLMs), it has also emerged as a promising tool for interpretability, suggesting the opportunity to understand model decisions through verbalized reasoning.
The paper investigates whether the tokens highlighted by large language models (LLMs) as key to their predictions are behaviorally consistent when those tokens are manipulated. Using sufficiency, comprehensiveness, and counterfactuality metrics, the authors evaluate closed‑source GPT‑4o‑mini and open‑source models Gemma3, Granite8B, and DeepSeek on IMDB and Steam review datasets. Results show that GPT‑4o‑mini consistently follows expected patterns, Gemma3 and Granite8B maintain sufficiency but falter under stricter interventions, and DeepSeek variants exhibit structural inconsistencies, indicating that explanation reliability varies across model families and sizes.
arXiv:2604.03754v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have been shown to encode truth of statements in their activation space along a linear truth direction. Previous...