arXiv:2606.05087v2 Announce Type: replace Abstract: Frequent verbs such as 'have' and 'make' can function either as collocates in light-verb constructions or as full lexical predicates, as in 'make a...
The paper investigates why AI-generated text has become harder to understand, focusing on psycholinguistic features linked to human processing difficulty such as object relative clauses, long syntactic dependencies, and low-frequency n-grams. Using a new corpus of AI-generated text from diverse domains, the study finds that frontier and near-frontier language models employ these difficult features more frequently than paraphrases from round-trip translation and human writing, including more inanimate subjects with animate verbs, longer dependencies, stacked nouns, and reduced object relatives. The authors hypothesize that AI models prioritize information density for tasks like math, code, and reasoning, diverging from human communication goals, and demonstrate this by fine‑tuning a model with a length penalty, which reduces human‑like linguistic behavior.
The paper investigates how large language models (LLMs) handle two usage‑based grammatical strategies: coercion, where a mismatched word is reinterpreted to fit its context, and preemption, where a plausible structure is avoided because it never appears in expected contexts. Experiments across models of varying sizes show that larger LLMs successfully recognize and produce constructional productivity in coercive situations, even with novel words, but they fail to exhibit preemptive behavior, lacking the negative evidence that would guide avoidance of unconventional but semantically viable constructions. The authors argue that coercion is detectable due to its surprising, semantically motivated nature, whereas preemption lacks such a signature, leading to an asymmetry where models coerce but do not preempt.
Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different...
The study investigates how augmenting training data with conceptually related alternatives—beyond nouns to include verbs and adjectives—affects large language models trained with next-token prediction. Results show that while concept augmentation consistently improves perplexity on news data regardless of part of speech, verbs yield the most reliable downstream performance gains. Representational analyses reveal that the conditions producing the largest downstream improvements do not always correspond to the greatest changes in internal representations.
The paper examines how language models learn action nominal constructions (ANCs) across 96 synthetic target languages with varied grammatical features. Using GPT‑2‑small trained from scratch, the study evaluates models on minimal pairs covering 34 grammatical phenomena, revealing that models mirror typological patterns in basic clause configurations but diverge in complement systems and ANC strategies. The findings suggest that natural language distributions may arise from both general learning pressures and functional demands not fully captured by current model experiments.
The study evaluates eleven autoregressive transformer models on English agreement attraction scenarios using a surprisal-based approach. Results show that while transformers match human reading times for prepositional phrase configurations, they perform poorly on object‑extracted relative clauses, with predictions diverging across models and failing to capture human interference patterns. The authors argue that current transformers cannot adequately model human morphosyntactic processing and call for more rigorous, comprehensive testing to avoid misleading conclusions from limited syntactic setups.
The study investigates whether pretrained transformer models encode functional words—such as pronouns and adverbs—in a way that mirrors human usage. By comparing embeddings of nouns with those of their functional counterparts in both isolated and parallel sentences, the authors find that functional words occupy a central yet distinct position in embedding space and that parallel lexicalized and functional sentences reside in different subspaces. Experiments show that only a mixed training set of functional and lexicalized sentences reveals shared syntactic and semantic structure, whereas training on either type alone fails to capture this parallelism.
The paper investigates whether a listener can infer a speaker’s intended meaning solely from the form of an utterance, using an information‑theoretic framework. By modeling language as a joint distribution over meanings, contexts, and utterances, the authors derive upper bounds on the probability that any text‑based featurizer—such as the hidden states of large language models—can recover the intended meaning. These bounds depend on the irreducible uncertainty left by form and the portion that can only be resolved by extralinguistic context, implying that no representation, regardless of data or supervision, can exceed these limits. Experiments on artificial languages, Mandarin zero‑pronoun resolution, and color reference support the theoretical predictions.
STRUCTURALCOST is a self‑paced reading dataset comprising 475 participants and 40,800 observations that isolates the processing cost of long‑distance subject‑verb dependency resolution. The study replicates a known psycholinguistic finding at NLP scale: human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance. Various language models—n‑gram, SSMs, and transformers—partially mirror this graded difficulty profile but consistently underestimate the integration cost humans incur, a gap that persists across architectures and model sizes.
arXiv:2609.37497v1 Announce Type: new Abstract: Modern transformer models excel at capturing semantic relationships through sentence embeddings, yet their ability to perform pragmatic reasoning remai...
arXiv:2610.07936v1 Announce Type: new Abstract: Systematicity, the probabilistic mapping of form to meaning, permeates language at all levels, and sublexical cues have been shown to govern human pseu...