The paper investigates whether a listener can infer a speaker’s intended meaning solely from the form of an utterance, using an information‑theoretic framework. By modeling language as a joint distribution over meanings, contexts, and utterances, the authors derive upper bounds on the probability that any text‑based featurizer—such as the hidden states of large language models—can recover the intended meaning. These bounds depend on the irreducible uncertainty left by form and the portion that can only be resolved by extralinguistic context, implying that no representation, regardless of data or supervision, can exceed these limits. Experiments on artificial languages, Mandarin zero‑pronoun resolution, and color reference support the theoretical predictions.
arXiv:2603. 18482v2 Announce Type: replace-cross Abstract: Standard decoding strategies for text generation, including top-$k$, nucleus sampling, and contrastive search, select tokens based on likelihood, restricting outputs to high-probability regions.
The paper argues that language operates with two parameters: amplitude, which measures how often words co‑occur, and phase, a signed relational factor that determines how co‑activated meanings combine and can reverse a meaning’s contribution. Unlike amplitude, phase is not captured by standard word embeddings or transformer attention weights and is indexed to individuals and dyadic interactions. The authors propose six empirical predictions to test phase’s role and suggest that future language models should incorporate agent‑indexed, phase‑bearing semantic states.
arXiv:2604. 14180v2 Announce Type: replace-cross Abstract: We train a 318M-parameter Transformer language model from scratch on a curated corpus of 1.
arXiv:2509.15114v2 Announce Type: replace Abstract: A controversial test for Large Language Models concerns the ability to discern possible from impossible language. While some evidence attests to th...
The paper investigates why AI-generated text has become harder to understand, focusing on psycholinguistic features linked to human processing difficulty such as object relative clauses, long syntactic dependencies, and low-frequency n-grams. Using a new corpus of AI-generated text from diverse domains, the study finds that frontier and near-frontier language models employ these difficult features more frequently than paraphrases from round-trip translation and human writing, including more inanimate subjects with animate verbs, longer dependencies, stacked nouns, and reduced object relatives. The authors hypothesize that AI models prioritize information density for tasks like math, code, and reasoning, diverging from human communication goals, and demonstrate this by fine‑tuning a model with a length penalty, which reduces human‑like linguistic behavior.
Language captures only a subset of the perceptions, intuitions and intents that form meaning between speakers. What, then, are the limits of language? We take steps towards measuring this in LLMs as f...
arXiv:2603. 20381v2 Announce Type: replace-cross Abstract: Understanding the fundamental mechanisms governing the production of meaning in the processing of natural language is critical for designing safe, thoughtful, engaging, and empowering human-agent interactions.
arXiv:2610.33433v1 Announce Type: new Abstract: Linguistic productivity, as evidenced by the use of words in novel contexts, is one of the most striking facts about human language. However, how abstr...
arXiv:2607.20995v2 Announce Type: replace Abstract: Distinguishing animate from inanimate concepts in written language requires more than shallow text processing, as it involves recognizing complex s...
The paper investigates the relationship between a large language model’s internal probability distribution and its verbalized confidence statements. By systematically manipulating training and in‑context data, the authors show that both internal and verbalized probabilities are influenced by distributional and asserted uncertainty in the data. They find that verbalized probabilities align with internal ones beyond what would be expected if they tracked the same sources independently, indicating that verbalized confidence can serve as a probe of the model’s internal distribution.
The study uses sparse autoencoders (SAEs) to probe how the Pythia-70M language model encodes grammatical number. By identifying and manipulating SAE‑derived features such as "plural noun" and "generate a plural verb," the authors show that removing these features forces the model to produce singular verbs, and vice versa. Similar asymmetries were observed in larger models (Gemma‑2‑2b and Llama‑3.1‑8b), indicating a privative representation of number in English.
The paper introduces Variance‑Calibrated Modulation (VCM), a training‑free pre‑decoding technique that reshapes language model probability distributions before truncation. VCM uses two dynamic mechanisms: a Contextual Searchlight via PMI to suppress stopwords and highlight context‑relevant tokens, and an Adaptive Self‑Debiasing that applies scale‑invariant penalization based on real‑time logit standard deviation. Experiments on open‑ended generation, factual QA, and mathematical reasoning show that VCM consistently reduces the likelihood trap, improving diversity, coherence, and reasoning accuracy with minimal computational cost.