arXiv AI

Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

arXiv AI
Oct 9

Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

The paper presents a white‑box deception detection system that uses probes trained on the largest deception dataset (FIBS) and a novel architecture aggregating across layers and tokens. The probes achieve 98.8% AUC on SHADE‑Arena, outperforming a text‑monitoring baseline, and reach up to 99.7% AUC when detecting introspective deception that cannot be inferred from context alone. They also effectively detect deception in open‑weight models on politically sensitive topics and when models are pressured about their beliefs.

By Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba, Adam Gleave, Chris Cundy
arXiv AI
Oct 1

Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations

The paper examines the reliability of lie detection probes for language models when the models adopt anti-factual personas, such as conspiracy theorists. A dataset of 8,916 human-reviewed responses from three LLMs was created, and eight existing probes were evaluated, revealing many fail to flag falsehoods under these personas. The authors also constructed confounder datasets showing that probes often track spurious correlations like instruction compliance, and propose a simple linear probe that performs best on both persona and confounder tests.

By Maximilian von Klinski, Sebastian Lapuschkin, Wojciech Samek, Lennart B\"urger
arXiv Computation and Language
Sep 4

Less Is Moral: A CHARMing Framework for Moral Foundations Detection in Endorsement Behaviour

The paper introduces CHARM, a lightweight fine‑tuned language model framework for detecting moral foundations in text. CHARM combines MAC cross‑attention, rationale alignment, and hate‑speech modulation to operationalize distinct psychological constructs, achieving up to 15.3% higher AUC in‑domain and outperforming supervised baselines on all out‑of‑domain datasets. The authors demonstrate CHARM’s scalability by applying it to large‑scale COVID‑19 Twitter data, revealing a strong link between moral value alignment and online endorsement behavior.

By Huixiang Fu, Marian-Andrei Rizoiu
arXiv Computation and Language
Sep 16

Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion

The paper introduces the SAST-IR framework to evaluate large language models’ robustness against persuasion attacks in a memory‑less setting, revealing a flaw called "Refusal Inertia" that masks true vulnerability. Using the CP‑Agent and a custom CounterFact‑Strict dataset, the authors demonstrate that simple, diverse attack strategies achieve a 96% success rate, while complex attacks often trigger defensive compliance. The study highlights severe brittleness in current state‑of‑the‑art models when deprived of conversation history.

By Zhuoang Cai