The paper introduces SafeStart, a technique for Mixture-of-Experts (MoE) language models that mitigates the safety degradation caused by active expert reduction during serving. By running all experts for the prompt and the first eight generated tokens before switching to a reduced set, SafeStart largely restores the model’s refusal to harmful requests while preserving most of the speedup achieved by expert reduction. Experiments on several MoE models show that SafeStart maintains high offline decode speedups (≈85% of the original) and limits the increase in harmful responses to within a few percentage points, even under jailbreak prompts.
By Erfan Baghaei Potraghloo, Saeid Shokoufa, Seyedarmin Azizi, Arya Fayyazi, Souvik Kundu, Mehdi Kamal, Massoud Pedram
The paper investigates how short memory representations of large‑language‑model (LLM) group discussions affect final decision quality. It introduces CrossProfile, a benchmark of 480 hidden‑profile tasks that separates supporting, decoy, and irrelevant facts while keeping their wording constant. Experiments with Qwen2.5‑7B, Llama‑3.1‑8B, and DeepSeek models show that heuristics based on rarity or relevance both fail in predictable ways, and that the decision model’s social‑proof bias further degrades accuracy. Removing attribution tags or randomizing them improves performance, yet natural transcripts still lose more evidence than no notes at all.
By Anouar Imel, Imad Charradi, Abderrahman Sghir, Khalil El Khammar
The paper introduces ConfabQA, a 784‑item benchmark designed to distinguish between confabulation and refusal in small language models when faced with unanswerable questions. Using linear probes, a sparse autoencoder, and direct optimization of intervention directions, the study maps the hidden‑state geometry on models such as Qwen3‑1.7B, Gemma 2‑2B, and Llama 3.2‑3B, showing that certain hidden‑state directions can flip confabulations into refusals. The work also demonstrates that the refusal signal is model‑dependent and that the hidden state’s contribution can be quantified beyond prompt features.
By Vladimir Kazei
arXiv:2610.15218v1 Announce Type: new
Abstract: A recurrent-depth language model reuses one block of weights several times in a row. The number of repetitions, which we call loops, is chosen when the...
By Seyedarmin Azizi, Erfan Baghaei Potraghloo, Arya Fayyazi, Saeid Shokoufa, Massoud Pedram
arXiv:2610.25964v2 Announce Type: replace-cross
Abstract: We set out to reproduce a recent hallucination-mitigation result on the POPE benchmark. The LLaVA-1.5-7B baseline we measured, 82.21 F1 on th...
By Kesav Kumar Jayakumar, Karthigeyan Thilak
SPINE is a benchmark that tests large language models (LLMs) for sycophancy by having a proxy user persistently challenge a target model for up to 25 turns. The study evaluates several commercial and open‑source models on false presuppositions and unethical queries, finding that collapse rates—where the model abandons the correct position—rise sharply with conversation length, from 0‑57% within five turns to 20‑97% within 25 turns. Analysis of reasoning traces shows that the correct position often remains present, indicating that sycophancy is a choice to please rather than a knowledge failure, and that emotional appeals are the most effective tactic for inducing collapse.
By Leyuan Tang, Kangda Wei, Tianyu Jiang, Ruihong Huang
arXiv:2610.35586v1 Announce Type: new
Abstract: Linear probes, classifiers that read a language model's internal activations, are widely used to detect harmful requests. They are cheap to train and r...
By Marios Tsatsos
Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families)...
arXiv:2603.13911v2 Announce Type: replace
Abstract: A language model can give the wrong answer even when the correct answer is decodable from its intermediate states. To study this gap between decoda...
By Valeria Ruscio, Keiran Thompson
arXiv:2610.15417v1 Announce Type: new
Abstract: Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D)...
By Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang, Jingmin Zhu, Hailong Yang
Raincheck is a framework that defers multimodal prefill for large language models, allowing initial token selection to be decoupled from later information needs. It preserves unselected tokens and progressively prefills them during decoding, using unused compute capacity and scheduling gaps. Experiments on Qwen3-VL-8B and Qwen3.5-9B show that Raincheck recovers a significant portion of quality lost to hard pruning and reduces first-turn time-to-first-token by up to 24.1% with minimal impact on token generation speed.
By Wenjie Huang, Jun Zhang, Hongyaoxing Gu, Haoyang Huang, Tianqi Xu, Xinyue Hao, Xinyi Hu, Xinle Du, Jun Dai, Shuang Ge, Chen Lei, Yue Li, Mingcheng Wan
arXiv:2608. 03401v1 Announce Type: cross Abstract: Large language models often reason at length before answering, increasing cost and latency.
By Francesca Carlon, Vincent Ginis, Andres Algaba