arXiv Machine Learning By Ibne Farabi Shihab, Sanjida Akhter, Md Najmus Swaqeeb, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Anuj Sharma

Spectral Tail Interventions in Decoder-Only Language Models: Reasoning-Sensitive Weight Structure from Controlled Surgery

Read the original on arXiv Machine Learning →

The paper investigates how the upper spectral tails of weight matrices in decoder‑only transformer language models influence reasoning behavior. By performing controlled interventions on the query–key product and comparing them to factor‑level surgeries, the authors find that edits targeting the spectral tail more strongly affect model performance across multiple checkpoints and reasoning benchmarks. The study also explores how inverse participation ratios predict accuracy transitions and shows that tail‑aware low‑rank adaptations converge faster than standard methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Oct 16

Amplitude-Only FFN Intervention for Tool-Structured LLM Inference: Method, Gated Evaluation Protocol, and Cross-Model Empirical Results

The paper introduces Amplitude Gating (AG), an inference-time technique that selectively masks feed‑forward network (FFN) activation coordinates while keeping pretrained weights unchanged. AG employs six SwiGLU sites to separate shared and gate‑only inputs, expands gate/value channels, and uses magnitude‑ranked entropy masks or a Newton‑Schulz windowed ranking variant to provide alternative forward paths without retraining. Experiments on three Qwen checkpoints and eight text/tool endpoints show modest improvements in function‑call and ToolBench endpoints, though statistical significance is not established and further experiments are required.

By Sheng Xu, Zhen Chen, Junhua Wang, Boyuan Huang, Ke Jia, Jiadun Zhu, Yiming Xu
arXiv Computation and Language
Oct 23

Spectral Smoothing and Spiking Perturbations Reconfigure Capability Profiles and Selective Task Performance in Large Language Models

The paper investigates how post‑hoc spectral smoothing and spiking interventions on transformer feed‑forward projection matrices affect large language models. By altering singular‑value spectra while preserving singular vectors and the geometric mean, the authors evaluate the impact on a range of tasks—including reasoning, truthfulness, hallucination, safety, and personality probes—across several Llama and Qwen models. The results reveal that structured perturbations can selectively suppress or enhance specific capabilities, suggesting a lightweight method for probing how different abilities are organized and traded off within these models.

By Qince Luan, Jianhui Xu, Yang Xu