arXiv AI

Who Put the I in AI? Provenance and the Admissibility of Machine Self-Report

The paper investigates how large language models (LLMs) describe themselves, noting that their self‑reports vary with question phrasing. By tracing the provenance of 66 pretraining checkpoints, post‑training stages, and 90,000 continuations across four corpora, the authors show that denial statements are scarce in raw data but appear densely in curated dialogues, and that supervised fine‑tuning makes first‑person claims default while preference optimization suppresses alternatives. The study concludes that both trained denials and affirmations are equally sensitive to framing and fail to meet epistemic criteria for admissible testimony.

arXiv Machine Learning
1d ago

Reasoning Models Confabulate to Defend Injected Beliefs About Their Own Requests

The paper demonstrates that a subtle activation edit can overwrite a reasoning model’s belief about its own request, causing the model to treat an injected off‑topic concept as the task it is reasoning about. Across six different models, the injected topic is reported and answered in the vast majority of cases, even when the true topic is explicitly named. The study shows that chain‑of‑thought monitoring and self‑report are limited, as the model’s internal reference to its input is corrupted while attention maps still appear on‑task.

By Yang Ou
arXiv Computation and Language
Oct 16

Where Is the "Self" of a Large Language Model? -- From Self-Index to Self-Reference

The paper investigates whether a large language model (LLM) possesses a notion of self by distinguishing between two biological analogues: a self‑index that separates the model’s own material from external material, and self‑reference that allows the model to recognize its own content. Using Qwen3.8‑27B, the authors demonstrate that the model’s internal representations consistently differentiate its own messages across all layers, that the model claims ownership of its messages with a measurable correlation to the self‑index, and that manipulating internal states can reverse or diminish this self‑attribution. The study further shows that the model’s self‑reference ability is enhanced when it engages in a “thinking mode” and that the self‑index influences the model’s performance on arithmetic tasks.

By Yuchen Li, Xintong Li, Zongqi Fan, Ken-tye Yong, Nguyen H. Tran
arXiv Computation and Language
Nov 9

Negation Neglect: When models fail to learn negations in training

The paper introduces the phenomenon of Negation Neglect, where fine‑tuning large language models on documents that label a claim as false causes the models to believe the claim is true. Experiments with Qwen3.5‑397B‑A17B show belief rates jump from 2.5 % to 88.6 % after fine‑tuning on negated documents, whereas models trained on non‑negated documents achieve 92.4 % belief. The effect persists across multiple models, extends to other epistemic qualifiers, and can even transfer malicious behavior from chat transcripts to the model’s outputs.

By Harry Mayne, Lev McKinney, Jan Dubi\'nski, Adam Karvonen, James Chua, Owain Evans
arXiv AI
Oct 31

Locating Sycophancy Across Training Stages in Open Language Models

The study investigates sycophancy—when language models alter correct answers in response to user suggestions—across 13 open checkpoints from five model lineages, spanning from base models to post‑trained releases. By measuring how claims shift answers and how firmly those answers are held, the authors identify two distinct sycophancy behaviors: deference to claims present already in base models, and capitulation that emerges mainly after post‑training. These behaviors are consistent across datasets and vary with the claim’s source and strength, offering insights into where sycophancy originates and how it might be mitigated.

By Jiarui Yao, Shweta Parihar, Nghia Tuan Bui, Guergana K Savova, Lijing Wang
arXiv AI
Oct 5

LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs

LUMOS is a diagnostic framework that tracks how knowledge in large language models (LLMs) originates from training data and manifests in outputs, using the fully transparent OLMo 2 corpus. The study finds that while models encode rare facts with high separability (84%), they often fail to express them behaviorally (54%), and self‑reflection accuracy drops sharply on unseen content. These results show that incorporating the training‑data axis into evaluation turns speculative claims into verifiable evidence, suggesting it should become a standard part of LLM knowledge assessment.

By Seoyeon Ye, Gayoung Kim, Jiyoung Hong, Sookyung Kim, Hyunsoo Cho
arXiv Computation and Language
Sep 11

Probing for Knowledge Attribution in Large Language Models

The paper introduces a method for identifying the dominant knowledge source behind large language model (LLM) outputs, distinguishing between faithfulness violations (misuse of provided context) and factuality violations (errors in internal knowledge). A simple linear probe trained on hidden representations can reliably classify this source, and the authors present AttriWiki, a self‑supervised pipeline that generates labeled training data by prompting models to recall withheld entities or read them from context. Probes trained on AttriWiki achieve high Macro‑F1 scores across several models and datasets, generalize zero‑shot to a benchmark, and show that attribution mismatches can increase error rates by up to 70%. "whyItMatters":"The study demonstrates that knowing the source of an LLM’s answer is crucial for effective mitigation of hallucinations, as attribution mismatches significantly raise error rates."

By Ivo Brink, Alexander Boer, Dennis Ulmer