arXiv:2610.18727v2 Announce Type: replace
Abstract: Natural Language Autoencoders (NLAs) are proposed as oversight tools for language models: an Activation Verbaliser (AV) writes an English explanati...
By Kevin Yandoka Denamgana\"i
arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?
By Ely Hahami, Ishaan Sinha, Lavik Jain
The paper introduces a multi‑signal pipeline for detecting hallucinations in large language models, combining fine‑tuned DeBERTa‑v3 classification, Monte Carlo Dropout uncertainty, and temperature‑scaled calibration. On the HaluEval benchmark it achieves high performance (F1 = 0.915, AUROC = 0.977) across QA, summarization, and dialogue, and shows that 25 % of training data yields 77 % of full‑data performance. The authors also demonstrate that applying Direct Preference Optimization to a Qwen2.5‑0.5B generator cuts hallucination rates from 85.5 % to 37.7 %, and that domain‑specific fine‑tuning (PubMedBERT on SciFact) outperforms general‑domain models for biomedical text.
By Varun Teja Chundru, Debasmita Biswas
arXiv:2607. 04223v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why.
By Mohamed Aly Bouke
The paper investigates how standard evaluations of detectors for hijacked large language model (LLM) agents may conceal important weaknesses. By auditing eleven evaluations on the AgentDrift and AgentDojo benchmarks, the authors show that detectors often perform poorly when faced with rare attacks, unseen tools, or data from a different benchmark, with precision dropping dramatically at low attack rates and rankings reversing as tool coverage falls. They also find that a released LLM action monitor, despite high AUROC on its own data, would still generate many false alerts when applied to a new benchmark.
"whyItMatters":"The study highlights that current evaluation practices can give a misleading sense of detector reliability, underscoring the need for more robust, cross‑benchmark testing to ensure real‑world safety of LLM agents."
By Mahmoud Abbasi, Juan M. N\'u\~nez V., Amin Shahraki
The paper proposes a training‑free detector that uses sentence‑level context sensitivity to identify unsupported content in retrieval‑augmented generation (RAG) answers. By re‑scoring each sentence with full context, no context, and each chunk removed, the method flags the chunk whose removal most reduces a sentence’s likelihood as the likely source. Evaluated on RAGTruth, TofuEval, and RAGBench, the detector outperforms answer‑level faithfulness scores, achieving AUCs up to 0.745 and matching per‑chunk fact‑checkers while using only a fraction of the compute required by large‑language‑model judges.
By Mohamed Aly Bouke