arXiv:2607. 22699v1 Announce Type: new Abstract: As Large Language Models (LLMs) grow more capable across diverse tasks, their (in)ability to generalize remains difficult to quantify and poorly understood beyond limited domains.
The paper investigates the intrinsic dimension (ID) of large language model (LLM) representations as an indicator of linguistic complexity. By comparing ID across model layers for coordination vs. subordination, right‑branching vs. center‑embedding, and unambiguous vs. ambiguous attachment, the authors find consistent ID differences that align with established complexity contrasts. Experiments across six LLMs, including representational similarity and layer pruning analyses, confirm that more complex phenomena produce higher ID profiles, with peaks occurring at different layers for each contrast.
arXiv:2610.14476v1 Announce Type: cross Abstract: Comparing large language models can reveal shared representational structure and systematic differences between them. However, aggregate similarity m...
The paper extends mechanistic interpretability of large language models by modeling concepts as low‑dimensional non‑linear manifolds rather than linear subspaces. It introduces a concept‑based alignment (CBA) score to compare these manifolds across layers and models, revealing block structures in intermediate layers, a shift from syntax‑dominated to mixed syntactic‑semantic concepts, and training‑dependent multilingual sharing. The study also shows that alignment patterns differ across model families and training stages, with adjacent stages aligning more closely than distant ones.
The paper introduces a framework to assess how linearly encoded linguistic relations are within language model embeddings. Using a constrained linear approximation on related and unrelated word pairs, the authors evaluate an extended BATS dataset across inflectional, derivational, lexicographic, and encyclopedic relations in GloVe, RoBERTa, and ModernBERT. Results show near-perfect linear encoding for inflectional and derivational relations, higher errors for lexicographic and encyclopedic relations—especially in one-to-many and many-to-many cases—and that RoBERTa and ModernBERT encode relations more linearly than GloVe.
Despite the success of neural models in natural language processing, their black-box nature limits interpretability and conceals the linguistic phenomena underlying their predictions. We present SLITE...
arXiv:2609.24932v1 Announce Type: new Abstract: Despite the success of neural models in natural language processing, their black-box nature limits interpretability and conceals the linguistic phenome...
The paper investigates how semantic meaning is arranged locally within large language models. By using two controlled probes—one cross‑lingual and one refinement probe—on four decoder‑only LLMs, it finds that after removing dominant variance directions, sentence pairs rank by semantic relatedness, semantic factors cluster around an invariant core with structured variation, refinement tightens neighborhoods, and sentence‑level versus token‑local readouts reveal complementary geometric patterns. These results suggest that LLM semantics form local anisotropic neighborhoods that contract when meaning is refined.
The study investigates how part‑of‑speech (PoS) categories are represented in the latent space of Sparse AutoEncoders (SAEs) applied to language models. Results show that PoS distinctions can be reliably recovered from SAE activations, but the mapping is not one‑to‑one; instead, PoS categories are supported by compact, distributed groups of sparse latents that vary across tags and remain stable on held‑out data. The findings suggest that SAEs encode morpho‑syntactic information in a distributed, category‑dependent manner rather than through isolated grammatical features.
The paper investigates whether language models can transfer hierarchical syntactic knowledge from a source language (L1) to a target language (L2) using a novel cross‑lingual poverty‑of‑stimulus (CPOS) experimental design. Results show that models can partially transfer L1 structural evidence to L2, producing a small but detectable hierarchical bias that varies across language pairs. The study further demonstrates that vocabulary overlap is crucial for successful transfer, indicating that cross‑lingual learning of hierarchical generalization is conditionally achievable.
The paper introduces a probe‑free method called the Neuron Separability Index (NSI) to assess how individual neurons in large language models distinguish grammatical from ungrammatical sentences using linguistic minimal pairs. Across 68 linguistic paradigms and seven model checkpoints, the study finds that while raw separability for morphology and syntax peaks early, single‑unit selectivity is sparse and weak, with rare strongly selective "grandmother neurons." Moreover, the research shows a dissociation between whole‑vector linear separability, single‑neuron selectivity, and behavioral competence, and demonstrates that targeted ablations can further separate activation selectivity from causal reliance.
The paper introduces an inductive method for learning concept embeddings from cross‑linguistic colexification networks combined with short textual descriptions. Unlike previous transductive approaches, this framework can embed unseen concepts using only a brief definition. The resulting embeddings correlate with cross‑linguistic similarity measures and generalize well to new concepts.