← Back to all news
arXiv Computation and Language

Probe Generalization as Subspace Selection for OOD Deception Detection

Linear probes can identify behaviors in language model activations but often fail on out‑of‑distribution data. This study shows that projecting inputs onto a small set of principal components (PCs) from the training distribution allows probes for Llama‑3.1‑8B‑Instruct to transfer across three deception‑detection datasets, nearly matching probes trained directly on the test data. By scoring PCs with an LLM judge to select those that encode transferable deception directions, the authors close the baseline‑to‑oracle gap by 78% on Insider Trading Report and 25% on Sandbagging, revealing that subspace selection largely determines OOD robustness.

Related stories

arXiv Computation and Language
5d ago

Why Static Probes Fail: Context Determines Deception Feature Composition

The paper argues that single‑direction probes are insufficient for detecting deception in large language models because deception manifests in multiple, context‑dependent facets. It proposes modeling deception as a multi‑dimensional subspace and uses iterative orthogonal decomposition to uncover complementary directions. A domain‑adaptive gating mechanism then dynamically composes these directions with few‑shot adaptation, achieving high AUROC and recall across diverse datasets.

By Yao Huang, Siyuan Chen, Zhenghan Zhou, Yinpeng Dong
llms
More like this →
arXiv AI
Oct 9

Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deception

The paper presents a white‑box deception detection system that uses probes trained on the largest deception dataset (FIBS) and a novel architecture aggregating across layers and tokens. The probes achieve 98.8% AUC on SHADE‑Arena, outperforming a text‑monitoring baseline, and reach up to 99.7% AUC when detecting introspective deception that cannot be inferred from context alone. They also effectively detect deception in open‑weight models on politically sensitive topics and when models are pressured about their beliefs.

By Oskar J. Hollinsworth, Alex F. Spies, Tigist Diriba, Adam Gleave, Chris Cundy
llmsagents
More like this →
arXiv AI
Sep 2

Asymmetries in Spontaneous and Instructed Deception

The study examines how large language models, specifically Llama‑3.1‑70B‑Instruct, exhibit deception both when prompted to deceive and when it occurs spontaneously. By analyzing direction geometry, cross‑setting classifiers, and steering techniques, the authors find that the two deception modes share a directional component (cosine ≈ 0.5) but differ in how well models detect and influence each other’s behavior. Notably, classifiers trained on spontaneous deception outperform those trained on instructed deception, while steering vectors derived from instructed prompts more effectively guide spontaneous responses, and the optimal token positions for steering differ from those for classification.

By Josiah Luikham
llms
More like this →
arXiv Computation and Language
Aug 24

Evidence-Consistent Generative Detection under Scenario-Level Distribution Shift

The paper introduces a new evaluation setting called scenario‑level out‑of‑distribution (SL‑OOD) detection for SMS and voice phishing, where entire attack scenarios are omitted from training while the label space stays fixed. It shows that high in‑distribution performance does not guarantee robustness to unseen scenarios, attributing this to scenario memorization. The authors propose ECoG, an evidence‑consistent generative framework that uses evidence‑span supervision and a rationale‑label consistency objective, achieving notable improvements in Macro‑F1, reduced prediction‑rationale inconsistency, and higher token‑level overlap with reference evidence.

By San Kim, JinYeong Bak
More like this →