Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
The Flow has not summarised this story yet — read it at arXiv AI.
arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
The paper investigates how sharing experts in mixture‑of‑experts (MoE) models across layers or computational steps can reduce local prediction error but may not always improve final outputs. By analyzing linear regression models, the authors derive criteria that balance variance reduction against bias introduced by sharing across roles with different target behaviors, and they identify conditions under which local gains persist in the final output. The study also provides error bounds for general routed computations, demonstrates that correct predictions alone do not confirm expert attribution, and presents controlled experiments showing cases where local improvements reverse into higher final‑output error.
The paper investigates how different optimizers acquire distinct generalizing relations in single‑hidden‑layer ReLU networks. By manipulating nuisance‑connected weights and performing a strict‑parity sweep, the authors show that training conditions reshape both shared failures and relative optimizer advantages, with specific interventions improving accuracy on hard stages. The study also compares optimizer behavior across tasks and an external architecture (CIFAR‑10).
arXiv:2608. 11233v1 Announce Type: cross Abstract: A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only annealing.
The paper investigates how synthetic tasks influence language‑model pretraining by distinguishing diagnostic, teachable, and transferable signals. In a controlled setting where 70% of the data is general Python and 30% is a mixture of curated synthetic tasks, literature‑derived probes, and OpenCodeInstruct, 14 of 27 tasks show teachability, with curated tasks outperforming literature probes. Curated synthetic data provides conditional value: it boosts downstream performance when mixed with OpenCodeInstruct but is less effective when used alone, and a loss‑based scheduler can misalign short‑term loss reduction with long‑term transfer.
arXiv:2610. 06950v1 Announce Type: cross Abstract: Injecting skills into a frozen language model currently costs a million parameters and a reinforcement-learning pipeline.