arXiv Machine Learning By Yang Ou

Reasoning Models Confabulate to Defend Injected Beliefs About Their Own Requests

Read the original on arXiv Machine Learning →

The paper demonstrates that a subtle activation edit can overwrite a reasoning model’s belief about its own request, causing the model to treat an injected off‑topic concept as the task it is reasoning about. Across six different models, the injected topic is reported and answered in the vast majority of cases, even when the true topic is explicitly named. The study shows that chain‑of‑thought monitoring and self‑report are limited, as the model’s internal reference to its input is corrupted while attention maps still appear on‑task.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Oct 23

Interpreting Model Reasoning through Behavioral Counterfactuals

The paper introduces a method for analyzing how interventions in a prompt affect a language model’s reasoning by using behavioral counterfactuals. It trains a module to edit the residual stream so that the next-token distributions for a fixed response match between a source and a counterfactual prompt, outperforming standard activation reconstruction. The resulting attribution maps and visualizations help identify causally important locations for counterfactual recovery and guide hypothesis testing with activation patching, as demonstrated in a case study on Gemma-4-E4B-it and DeepSeek-R1-0528-Qwen3-8B.

By Shawn Im, James Oldfield, Adel Bibi, Sharon Li
arXiv AI
Sep 25

Who Put the I in AI? Provenance and the Admissibility of Machine Self-Report

The paper investigates how large language models (LLMs) describe themselves, noting that their self‑reports vary with question phrasing. By tracing the provenance of 66 pretraining checkpoints, post‑training stages, and 90,000 continuations across four corpora, the authors show that denial statements are scarce in raw data but appear densely in curated dialogues, and that supervised fine‑tuning makes first‑person claims default while preference optimization suppresses alternatives. The study concludes that both trained denials and affirmations are equally sensitive to framing and fail to meet epistemic criteria for admissible testimony.

By Kristina \v{S}ekrst