arXiv:2607. 13425v1 Announce Type: cross Abstract: Learning effectively from limited data is critical in domains like security where labeled examples are scarce.
arXiv:2512. 20661v2 Announce Type: replace Abstract: Transformer-based pre-trained language models (PLMs) excel in text classification but suffer from attention dilution and attention sink effects, forcing models to over-focus on task-irrelevant tokens.
arXiv:2609.28117v1 Announce Type: cross Abstract: In this paper, we introduce a gradient-based head attribution strategy where the Token-level Max-Margin loss is backpropagated to the attention maps....
arXiv:2604. 22583v2 Announce Type: replace Abstract: Multi-head attention enables Transformers to capture diverse representations, but all attention heads are typically activated for every input, regardless of task complexity.
arXiv:2502. 08363v3 Announce Type: replace-cross Abstract: We present Top-Theta (Top-$\theta$) Attention, a training-free method for sparsifying transformer attention during inference.
arXiv:2609.35868v1 Announce Type: new Abstract: Is human readability necessary for effective fine-tuning of large language models? We investigate whether model-conditioned training representations ca...
arXiv:2512.20636v2 Announce Type: replace-cross Abstract: Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppr...
arXiv:2604. 01206v2 Announce Type: replace-cross Abstract: We present RELISH (REgression with a Latent Iterative State Head), a novel, lightweight architecture designed for text regression with large language models.
arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.
The paper introduces HeadEntropy, a training‑free method that predicts the correctness of large language model (LLM) answers by measuring how stable each attention head’s pattern is to further gradient updates. By linking the trace of the softmax Jacobian to 2‑Renyi entropy, the authors show that attention spread correlates with gradient stability, enabling accurate hallucination detection without reference annotations. Across five instruction‑tuned LLMs and five diverse datasets—including medicine, multi‑hop reasoning, and mathematics—HeadEntropy achieves a 0.736 AUROC, outperforming other training‑free baselines and matching hidden‑state probes while incurring less than 1% of inference cost.
arXiv:2504. 21174v2 Announce Type: replace Abstract: Deep learning drives a new wave in computing systems and triggers the automation of increasingly complex problems.
arXiv:2507. 01900v3 Announce Type: replace-cross Abstract: Pruning is a highly effective approach for compressing large language models (LLMs), significantly reducing inference latency.
arXiv:2607. 23054v1 Announce Type: cross Abstract: Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference.