The paper presents a method for distilling tabular foundation models (TFMs) into lightweight, dataset‑specific students. By using the full labeled training set as teacher context and training students on both observed and synthetic queries, the authors achieve significant performance gains over traditional supervised models on TabArena and TALENT benchmarks. The distilled students also provide substantial inference speedups, reducing the cost of repeated inference.
By Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo
arXiv:2607. 09375v1 Announce Type: new Abstract: We present Mach-Mind-4-Flash, a 35B-parameter Mixture-of-Experts (MoE) agentic model with 3B activated parameters.
By Foundation Model Team
arXiv:2608. 06137v1 Announce Type: new Abstract: Tabular data are ubiquitous in real-world applications and are crucial for data-driven prediction and decision-making across science, industry, finance, healthcare, and public services.
By Yi He, Zhengkang Guan, Anpeng Wu, Peng Cui, Fei Wu, Kun Kuang
arXiv:2605. 18383v2 Announce Type: replace Abstract: We present TabH2O, a foundation model for tabular data that performs classification and regression in a single forward pass via in-context learning.
By Pascal Pfeiffer, Dmitry Gordeev, Mathias M\"uller, Laura Fink, Joan Salv\`a Soler, Mark Landry, Branden Murray, Marcos V. Conde, Sri Satish Ambati
arXiv:2509. 14257v3 Announce Type: replace-cross Abstract: Large Language Model agents achieve strong performance on multi-step reasoning and tool-use tasks, but their impressive capabilities typically rely on extremely large backbones.
By Yuanjie Lyu, Chengyu Wang, Jun Huang, Tong Xu
arXiv:2606. 31048v1 Announce Type: cross Abstract: This paper investigates knowledge distillation from a large reasoning model (DeepSeek-R1) to a compact student model (Qwen2.
By Gaurab Baral, Aaditya Khanal, Yangyang Tao, Junxiu Zhou
ZGCM-1 is a 7B dense foundation model trained from scratch with extreme data, system, and algorithmic efficiency. It uses a core premise that compact models can overcome capacity limits by combining deliberate internal thinking with active external tool use, supported by a 256K context and an end‑to‑end high‑efficiency training recipe that includes interleaved gated sliding‑window and full attention, a stable FP8 Muon optimizer, progressive curriculum scaling, and reformulation of interaction traces into Markov Decision Processes. The model is competitive with much larger frontier models on challenging mathematical reasoning and agentic search tasks, offers a ~4.2× efficiency improvement in pre‑training time‑to‑loss, and its weights, checkpoints, training code, data recipes, and logs are fully open‑source to support community research.
By Jiyan He, Guang Liang, Hao Liu, Haoxiang Guan, Jinbo Sun, Junyi Guo, Wenjun Feng, Yantai Xie, Yifei Shen, Bin Shao, Chuyang Wei, Kai Chen, Kexin Zhou, Minghang Zhu, Shuxin Zheng, Tie-Yan Liu, Taine Zhao, Wenhui Zhu, Xueyin Xu, Xiaoqing Zhang, Yatao Li, Yuxuan Ren
The paper introduces MoEvolve, a bidirectional distillation framework that leverages increased activation budgets in Mixture-of-Experts (MoE) models to create a stronger internal teacher. By first restoring a high‑k model and then transferring its enhanced capabilities back to the original low‑k configuration, MoEvolve improves performance across various MoE models and benchmarks, achieving up to 22.2 domain‑average points and outperforming same‑budget RLVR baselines by up to 6.5 points. The resulting low‑k models often surpass their activation‑expanded teachers.
By Haoyu Chen, Jun Bai, Yang Liu, Jianfei Zhang, Wenge Rong
GEAR is a two‑stage framework that distills tabular foundation models into lightweight MLP or tree‑based predictors for efficient CPU deployment. In the first stage, synthetic covariates are used as teacher‑query locations to train the student on soft TFM targets, expanding coverage beyond observed rows. The second stage re‑anchors the student to the target distribution using real labels and out‑of‑fold teacher predictions, preventing self‑labeling leakage and improving performance. Experiments on TALENT and TabArena show that GEAR‑distilled MLPs outperform supervised MLPs by up to 2.00 AUC points on binary tasks and 1.35 on multiclass tasks, and also outperform CatBoost, while dramatically reducing inference time and memory usage.
By Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Yu Su, Peng Zhang, Ying Yan, Yifan Sun
The paper introduces SGUID, a method that selects a compact subset of skills for distillation in large language models. It shows that fewer than 25% of retrieved skills are useful for on‑policy distillation and that distilling only the selected skills can match or surpass full‑bank distillation while using up to 11× fewer skills. SGUID also enables stable model‑skill co‑evolution by iteratively curating new skill banks from updated model rollouts.
By Yuhan Liu, Xiyao Ma, Zhongkai Sun, Xu Han, Chengyuan Ma, Benjamin Z. Yao, Chenlei Guo
arXiv:2606. 12634v1 Announce Type: cross Abstract: Long-horizon tool-use reinforcement learning can learn from outcome verification, but its trajectory-level advantage is broadcast across many reasoning, API, and answer tokens.
By Tianyu Ding, Jianhong Xin, Juan Pablo De la Cruz Weinstein
arXiv:2606. 29280v1 Announce Type: cross Abstract: We identify intervention bias as a previously unquantified failure mode of zero-shot large-language-model (LLM) educational advisory agents: without task-specific training, they recommend action when a hindsight-optimal oracle policy mandates inaction.
By Craig Atkinson