Retrieval‑augmented generation (RAG) can expose language models to misleading evidence. In a study of 200 SciFact questions, the Gemma‑3‑12b model’s ability to detect poisoned claims dropped from 25% when a single poisoned document was present to 10% when six mutually agreeing poisoned documents dominated the context. The majority condition caused 34 of 38 questions to shift from detected to undetected, indicating a conformity‑like bias.
arXiv:2609.09243v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) grounds a language model in retrieved documents, which reduces hallucination but creates a new attack surface: i...
arXiv:2608.17153v3 Announce Type: replace Abstract: Retrieval-Augmented Generation (RAG) improves large language models by grounding them in external evidence, but this exposes them to knowledge-pois...
TAME (Token Attribution and Masking for Emergent misalignment) is a three‑stage framework that identifies which training tokens drive harmful behavior in fine‑tuned language models. It first scores tokens by how much fine‑tuning increases their likelihood, then characterizes patterns among high‑attribution tokens, and finally validates them by masking during training. Experiments on Llama and Qwen show that masking the top‑attribution tokens reduces emergent misalignment by 23‑ to 36‑fold, while random masking has no effect.
The paper examines how deliberative (System 2) reasoning affects a Retrieval-Augmented Generation (RAG) model’s vulnerability to knowledge‑poisoning attacks. Using two metrics—Cordon Rate and Leakage Rate—it evaluates six model configurations on 200 SciFact questions. Results show that enabling reasoning lowers both Cordon and Leakage Rates for DeepSeek‑V4‑Flash, indicating reduced behavioral impact from poisoned evidence, though overall attack success increases.
Fine-tuning an aligned language model on narrow, flawed data can induce harmful behavior far outside the training domain, known as emergent misalignment (EM). Prior work has localized EM in model weig...
arXiv:2608. 02678v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) systems are vulnerable to corpus poisoning: an attacker who inserts a crafted document into the retrieval corpus can steer the underlying large language model (LLM) toward an attacker-chosen wrong answer.
arXiv:2609.37616v1 Announce Type: new Abstract: Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on th...
arXiv:2610.34294v1 Announce Type: new Abstract: The stance expressed in responses from retrieval-augmented generation systems is significantly influenced by the stance distribution of the retrieved m...
arXiv:2610.28092v1 Announce Type: cross Abstract: Persuasive writing exploits cognitive biases, recruiting a reader's mental shortcuts to shape belief, often without the reader noticing. We call the...
The study investigates how false content in training data can influence language models’ downstream decisions even without explicit triggers. By comparing models trained on misleading versus truthful documents in a controlled decision task and a real‑world bushfire case, the authors find a gap between factual answers and the decisions derived from them: correct facts do not always lead to correct decisions, and removing an injected number does not eliminate the misleading narrative. The research highlights that data poisoning can subtly alter model behavior beyond what is detectable through direct probing.
Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News investigates whether people can distinguish news articles written by humans from those generated by large language models (LLMs). Using JudgeGPT, the study collected 2,318 judgments from 1,054 participants on content from six LLMs, finding that participants could not reliably differentiate machine-generated from human-written text. The research also shows that domain expertise predicts judgment accuracy, distinct response strategies exist, and accuracy declines after about 30 evaluations due to cognitive fatigue.
The paper introduces "stale‑document poisoning," a temporal alignment failure where outdated retrieval evidence causes retrieval‑augmented generation models to produce incorrect answers even when the model could answer correctly without retrieval. A benchmark of 317 verified knowledge reversals across medicine, law, software, and platform policy shows that outdated evidence flips 30–37% of Llama and Qwen answers, rising to 66–75% when models are explicitly instructed to trust the document. The study demonstrates that providing explicit validity information and a recency‑aware re‑ranker can substantially reduce poisoning, highlighting the need for models to assess whether retrieved evidence remains applicable.