The paper investigates whether features learned by sparse autoencoders (SAEs) in multiple‑choice question answering actually influence model predictions or merely correlate with errors. Using Gemma 3 1B and 4B models across eight datasets, the authors filter out SAE features that are tied to format or content artifacts and then perform causal interventions on the remaining features. They find that while SAE activations predict wrongness and uncertainty, causal interventions on filtered features shift answer margins more than random features but rarely flip answers, suggesting the features capture context‑dependent competition rather than the root cause of mistakes.
arXiv:2609.01322v1 Announce Type: cross Abstract: In many settings, studying causal questions based on text data requires adjusting for confounding information within texts. Yet there is a tradeoff i...
In many settings, studying causal questions based on text data requires adjusting for confounding information within texts. Yet there is a tradeoff in constructing text representations for adjustment:...
arXiv:2610.17898v1 Announce Type: cross Abstract: Sparse autoencoders decompose language model representations into interpretable features, but feature interpretation in this literature remains predo...
The paper investigates how visual interventions—training‑free edits applied to multimodal large language models—affect hallucinations and prediction quality. By applying a visual‑residual edit across successive decoder layers and matching its RMS strength, the authors identify an Evidence Anchoring Boundary (EAB) where the intervention’s influence drops sharply; effects before the EAB are 13.78–26.10× larger than after it. They further compare counterfactual subtraction across different question types and introduce calibrated selective routing, which only alters predictions when calibration supports an intervention, improving attribute accuracy by 1.56–1.81 percentage points while avoiding the harms of universal subtraction.
arXiv:2606. 08496v1 Announce Type: cross Abstract: Although Sparse Autoencoders (SAEs) have mitigated the opacity of large language models (LLMs) by decomposing dense representations into sparse features, explaining these features still remains a central challenge.
arXiv:2604. 26866v2 Announce Type: replace-cross Abstract: Large language models (LLMs) acquire most of their factual knowledge during the pre-training stage, through next token prediction.
arXiv:2609.37491v1 Announce Type: cross Abstract: Retrieval-augmented language models are expected to answer from the retrieved evidence, but in practice they often keep answering when that evidence...
arXiv:2609.08173v1 Announce Type: new Abstract: Sparse autoencoder (SAE)-based steering has been widely used to address knowledge conflicts by guiding LLMs to be more faithful to the contextual knowl...
The paper introduces RAG‑Stress, a diagnostic protocol that tests how retrieval‑augmented generation models handle misleading evidence. By fixing the question and reference answer, editing a single assertion to support an incorrect answer, and varying source‑priority policies and evidence positions, the study measures misleading rate (MR) and clean accuracy across fifteen systems on multiple QA datasets. Results show that prioritizing documents increases MR by 10.9–13.5 percentage points, with the highest MR when the misleading span appears at the end of the evidence.
arXiv:2605.16339v2 Announce Type: replace Abstract: Preference learning in large language models relies on reward models as proxies for human judgment. However, these models frequently exhibit prefer...
arXiv:2501. 06286v2 Announce Type: replace-cross Abstract: Multi-hop question answering requires a system to identify and integrate evidence distributed across documents, yet large language models remain vulnerable to irrelevant context.
The paper investigates why large language models hallucinate by proposing that failures often stem from inference misalignment rather than missing knowledge. It introduces a latent key-task model that shows pretraining-frequency imbalance can cause shortcut inference paths to dominate, leading to hallucinations. The authors create TrapQA, a diagnostic testbed with ScientistQA and Real-Life Constrained QA, to demonstrate two failure modes—task-retrieval bias and key-selection bias—where biased latent inference produces hallucinated answers.