arXiv Machine Learning

Characterizing Model-Native Skills

arXiv Machine Learning
Sep 30

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

arXiv:2609.37891v1 Announce Type: cross Abstract: Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines--...

By Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois, Pavel Chizhov, Carlos Rosas-Hinostroza, Neil Si Smail, Benjamin Burtin, Hanna Shcharbakova, Ivan Yamshchikov, Anastasia Stasenko
arXiv AI
Oct 1

Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions

Prompt2Skill is an unsupervised framework that constructs skills for Large Language Models directly from natural‑language task descriptions. It automatically derives task specifications, discovers or synthesizes datasets, and refines the skill through a reflective editing loop. In experiments across question answering, reading comprehension, spreadsheet manipulation, and mathematical reasoning, Prompt2Skill outperforms direct prompting, improving performance by an average of 10.8 points on both open‑source and frontier models.

By Bo Ni, Li Li, Ryan A. Rossi, Franck Dernoncourt, Tyler Derr
arXiv AI
Jun 26

NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research

arXiv:2606. 26671v1 Announce Type: new Abstract: Post-training alignment determines the reasoning and human preference following capabilities of large language models, yet most existing works withhold detailed data construction, filtering rules and training recipes, which hinders community reproducibility and lightweight model optimization.

By Qiaobo Hao, Yangqian Wu, Shunyi Wang, Zhongjian Zhang, Ziqun Li, Yayin He, Muqing Li, Chen Zhong
arXiv Computation and Language
Oct 23

Mimir: A Sub-2B Hierarchical Reasoning Model Trained on Permissible Data Punches Above Its Weight Class

Mimir is a 1.78‑B parameter language model built on the Hierarchical Reasoning Model Text (HRM‑Text) architecture, which interleaves low‑level and high‑level transformer stacks. It is trained from scratch on a curated mix of 161 instruction, reasoning, tool‑use, multilingual, and synthetic datasets totaling 70.5 B tokens per epoch (103 B for the continued‑training version Mimir v1.5), using only data that meets permissibility criteria. Across 20 benchmarks in English, Danish, and Math & Code, Mimir v1.5 outperforms all other sub‑2 B models and matches or exceeds the performance of larger 9 B‑parameter models, while memorisation audits show low risk and inference‑time ablations reveal graceful degradation when adjusting low‑level stack applications.

By Peter Schneider-Kamp, Jacob Nielsen, Gianluca Barmina, Danial Namazifard, Annemette Brok Pirchert, Kenneth Enevoldsen, Lukas Galke Poech
Hugging Face Trending Papers
Aug 9

SkillReason: Reasoning-Enhanced Agent Skill Retrieval for Implicit User Requests

Large language model agents increasingly rely on reusable skills to extend their capabilities beyond parametric knowl- edge. However, retrieving the appropriate skill from a large- scale library remains challenging because realistic user re- quests are often concise and underspecified, stating only the task goal while leaving the required capabilities and execu- tion steps implicit.

arXiv AI
Oct 12

BoundaryEvo: Data-Model Co-Evolution at the Capability Boundary for Long-Chain Reasoning Distillation

BoundaryEvo introduces a two‑stage data‑model co‑evolution framework for distilling long‑chain reasoning from large teacher models into small language models. First, Capability Priming distills a small set of teacher trajectories to give the student an initial reasoning baseline. Then, Capability‑Boundary Co‑Evolution iteratively selects questions the student has not mastered but the teacher can solve, updating the training data as the student’s capability grows. With only 1,000 examples, BoundaryEvo improves Qwen3.5‑4B accuracy from 62.9 % to 75.3 % on IMO‑AnswerBench and from 26.8 % to 46.0 % on AMO‑Bench, outperforming static and dynamic baselines across 11 benchmarks.

By Jiaming Su, Suorong Yang, Xudong Li, Tengchao Yang, Peng Ye
arXiv Machine Learning
Oct 16

AlignSAE: Concept-Aligned Sparse Autoencoders

AlignSAE is a method that aligns Sparse Autoencoder (SAE) features with a predefined ontology using a two‑stage curriculum: an initial unsupervised pre‑train followed by supervised post‑train that binds specific concepts to dedicated latent slots. This approach preserves general reconstruction capacity while creating an interpretable interface for inspecting and controlling individual concepts. Experiments on GPT‑2 show that AlignSAE enables precise causal interventions—such as reliable concept swaps with 94.8% success in a 1‑hop QA task—and achieves 9–∞× better disentanglement than standard SAEs, as measured by Mutual Information Gap, while supporting multi‑hop reasoning and probing generalization dynamics.

By Minglai Yang, Xinyu Guo, Zhengliang Shi, Jinhe Bi, Steven Bethard, Mihai Surdeanu, Liangming Pan