arXiv:2604. 22951v2 Announce Type: replace Abstract: Natural language data follows a power-law distribution, with most knowledge and skills appearing at very low frequency.
arXiv:2601. 22510v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often achieve strong benchmark accuracy yet remain brittle under small distribution shifts.
arXiv:2505.22635v2 Announce Type: replace-cross Abstract: A common approach for teaching large language models (LLMs) to reason is to train on chain-of-thought (CoT) traces of in-distribution reasoni...
The paper introduces a dependency‑graph framework to formalize compositional reasoning in language models, defining three increasing levels of compositionality. Using data‑structure tasks with deterministic rewards, the authors observe a consistent asymmetry: training on decomposed skills does not reliably transfer to composed tasks, whereas training on composed tasks transfers more readily to decomposed ones. They provide a theoretical explanation for this asymmetry and evaluate its effects under length extrapolation, structural distribution shift, and transfer to unseen skills, concluding with a pilot study on real‑world tool‑calling benchmarks that suggests the phenomenon extends to practical settings.
arXiv:2606. 07527v1 Announce Type: cross Abstract: The prevailing paradigm for training LLMs has evolved to rely on a massive post-training phase consisting of SFT and RL.
arXiv:2609.38764v1 Announce Type: new Abstract: Language models are typically pretrained from random initialization. Recent work challenges this convention, showing that a brief warm-up on abstract,...
The paper investigates data efficiency and selection in On‑Policy Distillation (OPD) for large language models. It shows that 1‑shot OPD—training on a single example—consistently improves performance, especially when the example is hard, and that longer chain‑of‑thought (CoT) paths drive the gains rather than token entropy. Based on these findings, the authors propose a simple hard‑example selection strategy that, using only eight carefully chosen hard examples, matches the performance of a 17,000‑example baseline across models from 1.5B to 7B parameters.
Large Language Models fail at implicit multi-hop reasoning: a model answers "When was $X$ born? " and "Who is $Y$'s closest friend?
arXiv:2602. 03685v2 Announce Type: replace-cross Abstract: Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable.
The paper investigates how large language models learn new tasks in-context, comparing rule-based instruction following to example-based few-shot prompting across five diverse tasks. Results show that models generally learn more reliably from rule descriptions than from examples alone, and adding more examples does not consistently improve performance. Instruction tuning further enhances rule-based learning while preserving example-based capabilities, with rule advantages being strongest for algebraic tasks and weaker for tasks requiring distributional sensitivity or parametric knowledge.
arXiv:2604.17614v2 Announce Type: replace-cross Abstract: Skills are a natural unit for describing what a language model can do and how its behavior can be changed. However, existing characterization...
arXiv:2602. 02470v2 Announce Type: replace Abstract: Autoregressive large language models (LLMs) have achieved remarkable success in many complex tasks, yet they can still fail in very simple logical reasoning such as the "reversal curse" -- when trained on forward knowledge data of the form "$A \rightarrow B$" (e.
arXiv:2609.38104v1 Announce Type: new Abstract: Power-sharpened sampling is an inference-time alternative to reinforcement-learning (RL) post-training for enhancing reasoning in large language models...