arXiv:2603.04317v2 Announce Type: replace-cross Abstract: A growing literature shows that variables can be linearly decoded from the activations of large language models (LLMs). These range from prop...
arXiv:2601. 11429v2 Announce Type: replace-cross Abstract: Hallucination is a central failure mode of language models (LMs).
The study investigates how well language models can generalize from fine‑tuned facts by distinguishing active facts—those that support further learning—from inert facts. Using a novel relabeling experiment (e.g., mapping Paris → Europe to Paris → crab), the authors fine‑tune models on a subset of subjects and test whether the learned mapping extends to unseen subjects sharing the same original answer. Across eleven models and 70 relations, they find a strong correlation (Pearson r = 0.87–0.92) between the linear readability of the original relation in pretrained representations and the model’s ability to generalize, a relationship that persists after controlling for various factors and remains robust to label randomization and semantic category changes.
Large language models are informing decisions with ever-higher stakes. As the consequences of their errors grow, a central question becomes harder to ignore: how much can we trust an individual answer...
arXiv:2607. 15495v1 Announce Type: cross Abstract: Out of everything the human brain processes, only a small fraction is consciously accessible, in the sense of being available for verbal report, deliberate control, and flexible reasoning.
The paper distinguishes between active facts, which help a language model generalize beyond mere recall, and inert facts, which do not. By relabeling answers to a known relation (e.g., Paris → Europe becomes Paris → crab) and fine‑tuning on a subset of subjects, the authors test whether the model can transfer the new label to unseen subjects sharing the same original answer. Across eleven models and 70 relations, the linear readability of the original relation in pretrained representations strongly predicts this generalization, with a Pearson correlation of 0.87–0.92, even after controlling for various factors.
The paper investigates when compressed vector representations can provide exact linear or affine readouts for a finite lexicon’s truth conditions, establishing a necessary and sufficient row‑space condition. It shows that the augmented truth matrix’s rank determines the minimal dimension needed for exact linear (rank r) and affine (rank r − 1) readouts, and that exact readouts preserve Boolean connectives. Experiments on GloVe and word2vec embeddings reveal that while many predicates are linearly separable, none achieves exact affine recovery from pretrained embeddings, yet supervised transductive training can attain exact affine recovery at dimensions meeting the theoretical bound, preserving most of the original variance. "whyItMatters":"The results provide a precise mathematical criterion for when vector embeddings can faithfully encode logical truth conditions, informing both theoretical understanding and practical training of language models."
arXiv:2610.09087v1 Announce Type: cross Abstract: Large language models are informing decisions with ever-higher stakes. As the consequences of their errors grow, a central question becomes harder to...
Relational BabyLM is a decoder‑only Transformer that replaces standard self‑attention with a Dual Attention Transformer (DAT) to separate object‑level lexical features from structural/relational information. The model incorporates a Next‑Latent Prediction objective to compress history into a dense belief state and introduces a RoPE‑based symbol‑retrieval mechanism. On the BabyLM 2026 challenge, the best model ranks 6th overall and 3rd on the NLP‑task subset, outperforming GPT‑2 on most benchmarks and achieving the highest EWoK score among strict‑track entries.
arXiv:2610.03551v1 Announce Type: new Abstract: Large language models (LLMs) have reached expert-level performance on competition mathematics largely through the volume of search placed around them:...
arXiv:2604.17614v2 Announce Type: replace-cross Abstract: Skills are a natural unit for describing what a language model can do and how its behavior can be changed. However, existing characterization...
arXiv:2608. 02412v1 Announce Type: new Abstract: Large language models (LLMs) have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data.