← Back to all news
arXiv Machine Learning By Francesco Karim Vicidomini

The Anatomy of a Truth Direction: Knowledge-Dependent Dimensionality, a Relational Law, and a Shared Category Geometry in Small Language Models

Read the original on arXiv Machine Learning →

The paper investigates how truth representations in small language models are structured. Using a training‑free axis derived from the dominant singular vector of hidden‑state differences between true and false minimal pairs, the authors evaluate 14 models across six architectural families, including Mixture‑of‑Experts. The study examines whether a single direction captures truth, which components contribute, and how this applies to categories with computed truth values.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
6d ago

Exact semantic readout from compressed vector representations

The paper investigates when compressed vector representations can provide exact linear or affine readouts for a finite lexicon’s truth conditions, establishing a necessary and sufficient row‑space condition. It shows that the augmented truth matrix’s rank determines the minimal dimension needed for exact linear (rank r) and affine (rank r − 1) readouts, and that exact readouts preserve Boolean connectives. Experiments on GloVe and word2vec embeddings reveal that while many predicates are linearly separable, none achieves exact affine recovery from pretrained embeddings, yet supervised transductive training can attain exact affine recovery at dimensions meeting the theoretical bound, preserving most of the original variance. "whyItMatters":"The results provide a precise mathematical criterion for when vector embeddings can faithfully encode logical truth conditions, informing both theoretical understanding and practical training of language models."

By Daniel Quigley
rag
More like this →
arXiv AI
Aug 28

How Language Models Organize and Structure Moral Knowledge

The study investigates how large language models encode moral knowledge by training linear probes for each category of Moral Foundations Theory. It finds that the model’s representations for different moral foundations occupy distinct, largely independent dimensions yet share a common positive component, indicating an integrated but nuanced moral structure. This geometry is consistent across model architectures and scales, emerges early in pre‑training, and reflects corpus statistics rather than the individualizing/binding distinction of the theory.

By Orion Reblitz-Richardson
llms
More like this →