arXiv AI
A Lie Detector Test for Language Models: Reading Knowledge a Model Won't Reveal
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 7
When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models
A 0.6B language model consistently answers YES to 1,200 logical tests, yet its behavior shows no discrimination. Linear probes reveal the correct verdict with high AUC (0.96) and transfer to unseen structures, but a single scalar readout fails due to a saturated decision threshold offset by +4.6 σ. Adjusting this threshold restores behavior accuracy from 50 % to 81 % and improves higher‑scale models, demonstrating that miscalibrated readouts, not hidden knowledge loss, drive performance gaps.
arXiv AI
Jul 23
Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations
arXiv:2607. 20379v1 Announce Type: new Abstract: Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it.
arXiv Machine Learning
Sep 11
Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
The paper investigates whether language models can identify sentences from their training data by using exact duplication counts from publicly released corpora for two model families, OLMo‑2 and Pythia. It finds that for typical duplication levels, models show only a weak trace of exposure, with a rank correlation near –0.08, and that strong signals only appear when a sentence appears roughly a thousand times, at which point fame rather than memory dominates. The study also demonstrates that common membership tests can be misleading, as changing a single word does not alter the model’s preference, and that controlling for register can significantly improve detector performance.
arXiv AI
Jun 17
Rift: A Conflict Signature for Deception in Language Models
arXiv:2606. 17229v1 Announce Type: cross Abstract: A model that lies while knowing the truth is the central case ELK cannot handle with behavioral evaluation alone.
arXiv Machine Learning
Aug 28
Can a Model Catch Its Own Hallucinations for Free?: Label-Free Doubt Signals Hold Their Own Against a Labelled Dataset for Abstention
The paper investigates whether a language model’s own confidence can replace labeled data for teaching it to abstain from uncertain answers. By fine‑tuning models with LoRA to answer only when their frozen confidence is high and to say “I’m not sure” otherwise, the authors show that this label‑free approach matches label‑supervised abstention tuning on short‑form factual QA. The method works across six open‑weight models (1B‑8B) and is effective except for confidently wrong facts, which the confidence signal cannot flag.
arXiv AI
Jun 12
"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
arXiv:2606. 12618v1 Announce Type: new Abstract: Robust lie detectors for language models could enable powerful techniques for auditing, monitoring, and post-hoc investigation of model behaviour, but evaluating them requires testbeds where models verifiably believe the opposite of what they say.