arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
By Zhiyan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, Jinqiao Wang
The paper investigates how sharing experts in mixture‑of‑experts (MoE) models across layers or computational steps can reduce local prediction error but may not always improve final outputs. By analyzing linear regression models, the authors derive criteria that balance variance reduction against bias introduced by sharing across roles with different target behaviors, and they identify conditions under which local gains persist in the final output. The study also provides error bounds for general routed computations, demonstrates that correct predictions alone do not confirm expert attribution, and presents controlled experiments showing cases where local improvements reverse into higher final‑output error.
By Tiantong Wang, Zhiren Gong, Yixin Wang, Tiantong Wu, Wei Yang Bryan Lim
The paper investigates how different optimizers acquire distinct generalizing relations in single‑hidden‑layer ReLU networks. By manipulating nuisance‑connected weights and performing a strict‑parity sweep, the authors show that training conditions reshape both shared failures and relative optimizer advantages, with specific interventions improving accuracy on hard stages. The study also compares optimizer behavior across tasks and an external architecture (CIFAR‑10).
By Gongyue Zhang, Honghai Liu
arXiv:2608. 11233v1 Announce Type: cross Abstract: A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only annealing.
By Mark Shapiro
The paper investigates how synthetic tasks influence language‑model pretraining by distinguishing diagnostic, teachable, and transferable signals. In a controlled setting where 70% of the data is general Python and 30% is a mixture of curated synthetic tasks, literature‑derived probes, and OpenCodeInstruct, 14 of 27 tasks show teachability, with curated tasks outperforming literature probes. Curated synthetic data provides conditional value: it boosts downstream performance when mixed with OpenCodeInstruct but is less effective when used alone, and a loss‑based scheduler can misalign short‑term loss reduction with long‑term transfer.
By Ohad Rubin
arXiv:2610. 06950v1 Announce Type: cross Abstract: Injecting skills into a frozen language model currently costs a million parameters and a reinforcement-learning pipeline.
By Ran Li, Lei Chen
The paper investigates Contrastive Activation Addition (CAA), a method that steers language‑model behavior by adding activation vectors derived from contrasting examples. By framing CAA as a behavior‑capability frontier, the authors select interventions that maximize behavioral gain while limiting capability degradation to specified budgets (1%, 3%, 5%). Across four instruction‑tuned models and seven behaviors, they find that validated behavioral effects generalize well, and that joint behavior‑capability survival improves significantly with higher budgets, though additional gains plateau. Control experiments confirm that the effects rely on learned behavior directions rather than generic perturbations, and representational geometry only modestly predicts cross‑behavior transfer. Secondary tests reveal that even interventions meeting protected benchmarks can still degrade untested capabilities or transfer unevenly to free‑form generation, underscoring that activation steering is a constrained, model‑dependent intervention rather than a universal control knob.
By Deepanshu Raj
LeafSkill introduces an external reuse layer for recursive LLM agents that retrieves semantically related action grounding experiences, binds action templates to current objects, and uses a lightweight 1.7B selector to choose or defer actions. This approach preserves high coverage of candidate actions under paraphrases (96–97%) and achieves a 97.34% adoption rate of unseen skill types. In experiments on Robotouille tasks, LeafSkill outperforms ReCAP, completing more tasks and reducing token usage and calls by over 14%.
By Shen Liu, Ze Zhou, Zhenguo Xu, Shaopu Wang, Yike Gao, Chunlei Wang
arXiv:2610.15492v1 Announce Type: new
Abstract: Skill internalization aims to convert externally specified procedural skills into persistent model capabilities, allowing language-model agents to exec...
By Haomiao Qiu, Yining Tang, Hongyi Wan, Changfeng Sun, Ziyue Qiao, Miao Zhang
arXiv:2601. 22108v2 Announce Type: replace-cross Abstract: Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoints, change data mixtures or objectives, and restart runs, while individual updates remain blind to target capabilities.
By Shuqi Ke, Giulia Fanti
arXiv:2608.29376v1 Announce Type: new
Abstract: Code generation increasingly relies on large transformer models, whose capability advances with scale. Yet such a scale is costly, creating demand for...
By Anjani Sirivella, Aanisha Newaz, Glaucia Melo
arXiv:2611.18417v1 Announce Type: cross
Abstract: Multimodal continual instruction tuning requires multimodal large language models to sequentially acquire new capabilities without catastrophic forge...
By Tao Hu, Ling-Xiang Zhang, Da-Wei Zhou