arXiv:2610.33025v1 Announce Type: cross Abstract: Letting a language model take gradient steps on the text it is reading, known as dynamic evaluation, lowers its loss on the rest of the document. Its...
arXiv:2607. 04223v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why.
The paper introduces FlashAttention with Statistics, a method that computes attention‑based detector features without materializing the full attention matrix, thereby reducing memory usage and latency. By accumulating statistics during the FlashAttention pass or via sparse lookups, the approach adds only a small overhead—16% at 2K tokens and 1.4% at 35K tokens on an H100 GPU—and stores just 22.5 KiB per response token. Experiments show that frozen Qwen3 proxy models, even as small as 0.6 B parameters, can serve as strong hallucination detectors, outperforming larger zero‑shot judges and specialized grounding checkers on several benchmarks, with attention features further boosting performance.
The paper shows that AI‑text detectors, rather than learning a clear AI‑versus‑human boundary, amplify an inherited predictability axis that already exists in language models. This amplification causes detectors to over‑flag fluent, formal human writing while missing high‑temperature AI outputs, and the bias persists across languages, code, and detector architectures. A training‑free operator can relocate the bias but cannot erase it, underscoring that the unfairness is a structural cost of out‑of‑distribution generalization.
arXiv:2608. 14712v1 Announce Type: cross Abstract: Each row of a transformer's attention matrix is a probability distribution over tokens, and in trained models most of that probability lands on a single \emph{sink} token, usually the first.
arXiv:2608. 02947v1 Announce Type: new Abstract: The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position.
The study investigates how post‑training quantization (PTQ) affects proactive interference (PI) in large language models. Using bitsandbytes, the authors compare FP16, INT8, and INT4/NF4 precision across three instruction‑tuned models and find that INT4 quantization markedly degrades accuracy under high interference, with INT8 also incurring a smaller penalty in two of the three models. The degradation is linked to increased same‑key intrusion errors and originates in the quantized transformer backbone rather than the output layer.
The paper uncovers a systematic mismatch in modern text embeddings: while sentiment can be extracted with high accuracy from frozen encoders, the embeddings’ cosine similarity metric barely reflects polarity differences. Across eleven models from 2021–2026, a linear probe achieves up to 91.3% accuracy, yet cosine similarity margins are only 0.003–0.064, leading to poor retrieval performance (e.g., Qwen3-Embedding-4B’s 64.3 MAP). Introducing a lightweight supervised‑contrastive head (Sentiment‑Injected Sentence Embeddings) dramatically improves probe accuracy, retrieval MAP, and polarity margins, but at the cost of downstream semantic similarity. "whyItMatters":"The findings reveal that widely used retrieval and clustering systems may ignore valuable sentiment information encoded in embeddings, impacting their effectiveness."
The paper investigates whether language models can identify sentences from their training data by using exact duplication counts from publicly released corpora for two model families, OLMo‑2 and Pythia. It finds that for typical duplication levels, models show only a weak trace of exposure, with a rank correlation near –0.08, and that strong signals only appear when a sentence appears roughly a thousand times, at which point fame rather than memory dominates. The study also demonstrates that common membership tests can be misleading, as changing a single word does not alter the model’s preference, and that controlling for register can significantly improve detector performance.
arXiv:2608. 04678v1 Announce Type: cross Abstract: Papers 1-2 of the Kathleen series showed that a byte-level, attention-free architecture built from a wavetable encoder and multi-scale reverberant state can match strong baselines on classification at ~450-700K parameters, without pretraining.
The paper proposes a training‑free detector that uses sentence‑level context sensitivity to identify unsupported content in retrieval‑augmented generation (RAG) answers. By re‑scoring each sentence with full context, no context, and each chunk removed, the method flags the chunk whose removal most reduces a sentence’s likelihood as the likely source. Evaluated on RAGTruth, TofuEval, and RAGBench, the detector outperforms answer‑level faithfulness scores, achieving AUCs up to 0.745 and matching per‑chunk fact‑checkers while using only a fraction of the compute required by large‑language‑model judges.
arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?