The paper investigates how the choice of Transformer layers for fine‑tuning affects both task performance and the preservation of pre‑trained capabilities. By measuring layer sensitivity with a forward‑only input–output cosine similarity metric, the authors identify layers that are more suitable for adaptation. They introduce Layer‑Selective LoRA (LS‑LoRA), which trains adapters only in low‑similarity layers, and demonstrate that this approach improves target‑task results while better retaining commonsense reasoning compared to standard all‑layer LoRA.
arXiv:2602. 05988v2 Announce Type: replace Abstract: Pre-training Large Language Models (LLMs) on web-scale datasets becomes fundamental for advancing general-purpose AI.
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
arXiv:2604. 26170v2 Announce Type: replace Abstract: Adapting large language models (LLMs) to a targeted task efficiently and effectively remains a fundamental challenge.
arXiv:2602. 14696v2 Announce Type: replace Abstract: Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task.
Large language model agents increasingly rely on reusable skills to extend their capabilities beyond parametric knowl- edge. However, retrieving the appropriate skill from a large- scale library remains challenging because realistic user re- quests are often concise and underspecified, stating only the task goal while leaving the required capabilities and execu- tion steps implicit.
arXiv:2608. 08640v1 Announce Type: new Abstract: Large language model agents increasingly rely on reusable skills to extend their capabilities beyond parametric knowl- edge.
arXiv:2604. 24594v3 Announce Type: replace-cross Abstract: As large language models (LLMs) evolve into agentic problem solvers, they increasingly rely on external, reusable skills to handle tasks beyond their native parametric capabilities.
arXiv:2608.30462v1 Announce Type: cross Abstract: Large language models exhibit substantial performance variation across languages, even when solving semantically equivalent tasks. Existing analyses...
arXiv:2608. 06880v1 Announce Type: new Abstract: General-purpose skills promise reusable procedural knowledge for language agents, yet semantic relevance does not guarantee execution utility: a retrieved skill may encode assumptions that conflict with the current task, execution environment, or other retrieved skills.
Agent skills are increasingly used to equip large language model (LLM) agents with reusable procedural knowledge. Although recent work has substantially improved skill retrieval due to the increasing skill libraries, retrieving a plausible skill bundle does not guarantee that executing it is worthwhile.
The paper investigates how small language models acquire abstract reasoning skills on the ARC‑TGI benchmark, which groups grid‑transformation tasks into controllable families and allows resampling, spatial shifts, and cross‑benchmark transfer. Over 1,000 supervised fine‑tuning runs across decoder‑only, encoder‑decoder, and mixture‑of‑experts families, the study finds that high in‑distribution accuracy is possible but depends heavily on optimization and is uneven across task families. Performance drops sharply outside the training distribution, and gains from larger training sets or additional in‑context examples vary by model family; attention diagnostics reveal distinct patterns but do not explain causal mechanisms.
The paper introduces ATLAS, a method that uses retained-domain activation atlases to guide task-specific fine‑tuning of language models. By providing local reference centers and directional filters, ATLAS learns a low‑rank residual that adapts to new tasks while preserving existing behavior. Experiments on Qwen3‑8B and other backbones show lower retained‑output KL divergence, fewer rewritten answers, and more stable responses compared to seven baselines.