arXiv:2608. 08107v1 Announce Type: cross Abstract: Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining.
arXiv:2606. 26620v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a powerful tool for decomposing superposed language model representations into sparse and interpretable features.
arXiv:2606. 27786v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) enhances LLMs by incorporating external knowledge to support response generation.
The paper introduces Retrieval-Augmented Decoding (RAD), a decoding-time method that improves the truthfulness of large language models without retraining. RAD uses a small reference set of up to ten annotated examples to build a grounding space of context embeddings and next-token logits, which it retrieves and aggregates during inference to shape the model’s output. Experiments on four open-ended generation benchmarks and four different LLMs show that RAD consistently outperforms strong baselines and generalizes well across tasks.
The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.
arXiv:2510. 18940v2 Announce Type: replace-cross Abstract: Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into two categories: addition-based and selective in-situ adaptation.
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
Task-Level Natural Language Priors as Learning Signals for Low-Resource LLM Training proposes Prior-Guided Tuning (PGT), a training approach that treats natural-language priors as auxiliary learning signals rather than just input context. The method introduces Contrastive Prior Steering (CPS), which adds positive and negative prior-conditioned auxiliary losses while preserving the original supervised objective. Experiments on AmbiMath, Jigsaw, and MNLI/HANS demonstrate that CPS consistently outperforms plain and prompt fine-tuning, achieving high accuracy and significant gains with limited training data.
The paper introduces a family of adapters that enhance language model reasoning by adding selective state-space control at token and context levels. The token-level MaLoRA makes the adapter’s scaling factor dynamic and recurrent, improving over static low‑rank adaptation. The context-level MaRA tracks cross‑segment reasoning state and retrieves relevant segments, outperforming an eight‑billion‑parameter dense retriever and boosting reasoning accuracy by an average of +6.4 F1 over LoRA.
arXiv:2507. 04221v3 Announce Type: replace-cross Abstract: We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of large language models (LLMs) without weight updates.
arXiv:2609.35868v1 Announce Type: new Abstract: Is human readability necessary for effective fine-tuning of large language models? We investigate whether model-conditioned training representations ca...
arXiv:2610.01054v1 Announce Type: cross Abstract: In-context learning (ICL) enables language models to perform new tasks from demonstrations without weight updates. However, every ICL inference requi...