The paper investigates how large language models (LLMs) handle two usage‑based grammatical strategies: coercion, where a mismatched word is reinterpreted to fit its context, and preemption, where a plausible structure is avoided because it never appears in expected contexts. Experiments across models of varying sizes show that larger LLMs successfully recognize and produce constructional productivity in coercive situations, even with novel words, but they fail to exhibit preemptive behavior, lacking the negative evidence that would guide avoidance of unconventional but semantically viable constructions. The authors argue that coercion is detectable due to its surprising, semantically motivated nature, whereas preemption lacks such a signature, leading to an asymmetry where models coerce but do not preempt.
By Claire Bonial, Claire Benet Post, Laura Michaelis, Harish Tayyar Madabushi
The paper investigates natural language autoencoders (NLAs) as tools for verbalizing activation differences in large language models. It examines whether NLAs capture compositional structure, generalize to fine‑tuned models, recover fine‑tuning objectives, and can be specialized for model diffing. Experiments across four models show that NLAs largely succeed in these tasks, though they do not yet match state‑of‑the‑art diffing methods on emergent misalignment detection.
By Andrea Blasi N\'u\~nez, Gianluca Barmina, Federico Torrielli, Thiusius Rajeeth Savarimuthu, Peter Schneider-Kamp, Lukas Galke Poech
The paper investigates natural language autoencoders (NLAs) that convert large language model activations into readable text. It examines whether NLAs capture compositional structure, generalize to fine‑tuned models, recover fine‑tuning objectives, and can be specialized for model diffing. Experiments across multiple models show positive results: NLAs preserve compositionality, transfer to lightweight fine‑tunes, recover hidden objectives, and outperform a black‑box baseline on diffing tasks, though they lag behind state‑of‑the‑art methods for misalignment detection.
arXiv:2606. 00230v1 Announce Type: new Abstract: Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs.
By Sherin Muckatira, Namrata Shivagunde, Vijeta Deshpande, Anna Rumshisky
The paper investigates whether language models can transfer hierarchical syntactic knowledge from a source language (L1) to a target language (L2) using a novel cross‑lingual poverty‑of‑stimulus (CPOS) experimental design. Results show that models can partially transfer L1 structural evidence to L2, producing a small but detectable hierarchical bias that varies across language pairs. The study further demonstrates that vocabulary overlap is crucial for successful transfer, indicating that cross‑lingual learning of hierarchical generalization is conditionally achievable.
By Shin Yamaguchi, Masaya Taniguchi, Tatsuki Kuribayashi, Taro Watanabe, Yoshinobu Kano
The study investigates how augmenting training data with conceptually related alternatives—beyond nouns to include verbs and adjectives—affects large language models trained with next-token prediction. Results show that while concept augmentation consistently improves perplexity on news data regardless of part of speech, verbs yield the most reliable downstream performance gains. Representational analyses reveal that the conditions producing the largest downstream improvements do not always correspond to the greatest changes in internal representations.
By Laya Iyer, Chen Shani