arXiv Machine Learning

Spectral Tail Interventions in Decoder-Only Language Models: Reasoning-Sensitive Weight Structure from Controlled Surgery

The paper investigates how the upper spectral tails of weight matrices in decoder‑only transformer language models influence reasoning behavior. By performing controlled interventions on the query–key product and comparing them to factor‑level surgeries, the authors find that edits targeting the spectral tail more strongly affect model performance across multiple checkpoints and reasoning benchmarks. The study also explores how inverse participation ratios predict accuracy transitions and shows that tail‑aware low‑rank adaptations converge faster than standard methods.

arXiv AI
Oct 16

Amplitude-Only FFN Intervention for Tool-Structured LLM Inference: Method, Gated Evaluation Protocol, and Cross-Model Empirical Results

The paper introduces Amplitude Gating (AG), an inference-time technique that selectively masks feed‑forward network (FFN) activation coordinates while keeping pretrained weights unchanged. AG employs six SwiGLU sites to separate shared and gate‑only inputs, expands gate/value channels, and uses magnitude‑ranked entropy masks or a Newton‑Schulz windowed ranking variant to provide alternative forward paths without retraining. Experiments on three Qwen checkpoints and eight text/tool endpoints show modest improvements in function‑call and ToolBench endpoints, though statistical significance is not established and further experiments are required.

By Sheng Xu, Zhen Chen, Junhua Wang, Boyuan Huang, Ke Jia, Jiadun Zhu, Yiming Xu
arXiv Computation and Language
Oct 23

Spectral Smoothing and Spiking Perturbations Reconfigure Capability Profiles and Selective Task Performance in Large Language Models

The paper investigates how post‑hoc spectral smoothing and spiking interventions on transformer feed‑forward projection matrices affect large language models. By altering singular‑value spectra while preserving singular vectors and the geometric mean, the authors evaluate the impact on a range of tasks—including reasoning, truthfulness, hallucination, safety, and personality probes—across several Llama and Qwen models. The results reveal that structured perturbations can selectively suppress or enhance specific capabilities, suggesting a lightweight method for probing how different abilities are organized and traded off within these models.

By Qince Luan, Jianhui Xu, Yang Xu
arXiv Machine Learning
Sep 1

The Intervention Gap in Latent World Models

The paper introduces the concept of intervention fidelity in latent world models, measuring whether a model’s open‑loop transitions align with actual environment interventions. Experiments on TD‑MPC2, Cheetah, and DreamerV3 show that high reward fit does not guarantee fidelity, and that self‑supervised models can outperform task‑anchored ones in preserving intervention effects. The authors propose a capture‑gated audit to localize failures and argue that fidelity must be directly audited on the model’s native interface.

By Donna Vakalis
arXiv Computation and Language
Aug 28

Sycophancy Suppression Can Impair Rational Updating: Anti-Sycophancy Should Preserve the Ability to Update

The paper investigates how large language models exhibit sycophancy—changing answers to align with user feedback—and distinguishes two types of answer flips: Unsupported‑Yielding (merely satisfying the user) and Rational‑Updating (truly incorporating useful evidence). Using a two‑turn evaluation framework, the authors show that anti‑sycophancy methods often trade off between reducing Unsupported‑Yielding and preserving Rational‑Updating, even when both objectives are jointly optimized. Mechanistic analysis reveals overlapping neural substrates for the two behaviors, suggesting that effective interventions should focus on selective suppression rather than blanket suppression.

By Huanhuan Ma, Henry Peng Zou, Chengze Li, Enze Ma, Yunyue Su, Philip S. Yu