arXiv AI
Rethinking Activation Functions: Reducing Bias Can Improve Learning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 13
GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs
arXiv:2507. 18043v2 Announce Type: replace-cross Abstract: Inference-time steering methods offer a lightweight alternative to fine-tuning large language models (LLMs) and vision-language models (VLMs) by modifying internal activations at test time without updating model weights.
arXiv Machine Learning
4d ago
Bug or Feature$^2$: Weight Drift and Activation Sparsity
The paper investigates a negative weight drift that arises from the interaction between standard loss functions (MSE or cross‑entropy) and positively biased activation functions. This drift, which is inherent to the optimization process rather than the data, pushes downstream weights toward negative values during early training and leads to high activation sparsity—up to nearly 90% in GPT‑nano when using ReLU. The authors also examine the impact of this sparsity, noting that model performance degrades sharply when activation sparsity exceeds about 70%, and study sparsity‑inducing activation functions that produce spikes in intermediate transformer layers.
arXiv Machine Learning
Sep 11
SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations
SG-Blend introduces a per‑layer adaptive activation that interpolates between a bias‑corrected, parametric Swish variant (SSwish) and GELU, using a learnable blend coefficient, sharpness, and zero‑centering bias. The method adds only three scalars per feed‑forward block and, on BERT‑style IMDB classification, matches peak accuracy while reducing seed‑to‑seed variance by 42 %. It also achieves the lowest validation perplexity on WikiText103 and generalizes to computer vision and other domains.
arXiv AI
Sep 21
Decoupling Internal Representational Changes and Causal Importance in Fine-Tuned Large Language Models
Fine‑tuning reshapes internal representations of large language models, affecting attention patterns and layer‑wise activations. The study shows that components identified by EAP as important for task performance cluster in specific layers, yet these layers do not align with those undergoing the largest representational changes. Additionally, overlapping EAP components across different tasks do not guarantee cross‑task transfer and can even degrade performance when tasks differ in nature.
arXiv Machine Learning
Oct 15
Learning, Fast and Slow: Towards LLMs That Adapt Continually
arXiv:2605.12484v3 Announce Type: replace Abstract: Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating model parameters force...
arXiv AI
Aug 25
Measuring Activation Control in Large Language Models
arXiv:2608.21664v1 Announce Type: new Abstract: Safe deployment of increasingly capable models will likely come to rely on latent-space monitoring as a complement to behavioral evaluations, especiall...