PTXBench is a benchmark designed to evaluate and adapt large language models (LLMs) for GPU kernel optimization using architecture-specific PTX code. It assesses functional correctness, runtime execution of target instructions, and speedup over leading libraries on GEMM and attention workloads on H100 and B200 GPUs. The study finds uneven performance across models, especially on complex attention backward tasks, and shows that fine‑tuning Qwen3.6‑27B improves some tasks but generalization remains inconsistent.
The paper introduces Ladder Side Tuning (LST), a parameter‑efficient fine‑tuning method that adds a lightweight side network to large language models. LST matches QLoRA’s compute scaling while halving peak memory usage, enabling 7B‑parameter models to be fine‑tuned on a single 12 GB GPU with 2k‑token contexts without gradient checkpointing. The authors also present xLadder, a depth‑extended variant that increases effective depth through cross‑connections, allowing deeper reasoning without extra memory overhead.
By Estelle Zheng, Nathan Cerisara, S\'ebastien Warichet, Emmanuel Helbert, Christophe Cerisara
arXiv:2606. 04023v1 Announce Type: cross Abstract: While large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments (e.
By Jie Li, Wenzhao Wu, Junqi Hu, Qinrui Zheng, Bowen Wu, Juepeng Zheng, Yutong Lu, Haohuan Fu
arXiv:2605. 19276v3 Announce Type: replace-cross Abstract: In recent years, the field of artificial intelligence has undergone a paradigm shift from task-specific small-scale models to general-purpose large language models (LLMs).
By Maosong Cao, Kai Chen, Haodong Duan, Yixiao Fang, Zhiwei Fei, Tong Gao, Ge Jiaye, Mo Li, Hongwei Liu, Junnan Liu, Yuan Liu, Chengqi Lyu, Han Lyu, Ningsheng Ma, Zerun Ma, Yu Sun, Zhiyong Wu, Linchen Xiao, Zhuozhi Xiong, Jun Xu, Haochen Ye, Zhaohui Yu, Yike Yuan, Songyang Zhang, Yufeng Zhao, Fengzhe Zhou, Peiheng Zhou, Dongsheng Zhu, Lin Zhu, Jingming Zhuo
PTXBench is a benchmark designed to evaluate and adapt large language models (LLMs) for GPU kernel optimization using architecture‑specific PTX code. It assesses functional correctness, runtime execution of target instructions, and speedup over state‑of‑the‑art libraries on GEMM and attention workloads on H100 and B200 GPUs. The study finds uneven success rates, especially on complex attention backward tasks, and shows that executing target instructions does not guarantee competitive performance, with no model consistently outperforming frontier libraries. The authors also fine‑tune Qwen3.6‑27B, noting that repair‑conditioned training improves some tasks but generalization remains inconsistent, highlighting the importance of data coverage, balance, and teacher quality.
By Genghan Zhang, Yixin Dong, Chengze Fan, Zhichen Zeng, Yueming Yuan, Shaowei Zhu, Kunle Olukotun
arXiv:2603. 29002v3 Announce Type: replace-cross Abstract: Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning.
By Zifan He, Rui Ma, Yizhou Sun, Jason Cong
Instella‑MoE is a fully open Mixture‑of‑Experts language model with 16 billion total parameters and 2.8 billion active parameters per token, trained from scratch on AMD Instinct GPUs. It incorporates a sparsely activated MoE design with Gated Multi‑head Latent Attention and FarSkip‑Collective connectivity, and follows a multi‑stage pipeline that includes pre‑training, long‑context extension, supervised fine‑tuning, direct preference optimization, and reinforcement learning with Multi‑Teacher On‑Policy Distillation. The model achieves an average score of 76.7 on pre‑training benchmarks and 73.2 on instruction‑following, reasoning, math, coding, and chat benchmarks, outperforming comparable fully open and open‑weight models, and its full training pipeline, weights, and code are released for reproducibility.
By Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Zicheng Liu, Emad Barsoum
arXiv:2608. 12004v1 Announce Type: cross Abstract: In modern AI frameworks, GPU kernels are key to overall system performance.
By Jinjun Huang, Zhongzhen Wen, Tongtong Xu, Meng Yan, Xin Xia, Zhongxin Liu
arXiv:2606. 04847v1 Announce Type: cross Abstract: Native GPU kernel generation turns high-level tensor programs into executable, efficient low-level code.
By Kun Cheng, Songshuo Lu, Sicong Liao, Tankun Li, Yafei Zhang, Dong Yang, Qiheng Lv, Hua Wang, Zhi Chen, Yaohua Tang
arXiv:2606. 30062v1 Announce Type: cross Abstract: While large language models have been dominating the research landscape recently, small language models remain highly relevant across various domains; yet, they receive far less attention.
By Dari Baturova, Elena Bruches, Ivan Chernov, Roman Derunets, Arsenii Fomin, Andrey Kostin
Agnostics is a language‑agnostic post‑training pipeline that uses reinforcement learning with verifiable rewards (RLVR) to improve large language models on low‑resource programming languages. By rewriting unit‑test datasets into a language‑independent I/O format, providing a short configuration for compiling and running code, and employing a single verifier that judges code by observable behavior, Agnostics eliminates the need for language‑specific engineering. Applied to Lua, Julia, R, OCaml, and Fortran, it boosts Qwen‑3 4B to rival larger models, scales to diverse families, and achieves new state‑of‑the‑art pass@1 on MultiPL‑E and a new multi‑language LiveCodeBench.
By Aleksander Boruch-Gruszecki, Yangtian Zi, Zixuan Wu, Tejas Oberoi, Carolyn Jane Anderson, Joydeep Biswas, Arjun Guha
arXiv:2608. 19889v1 Announce Type: new Abstract: The entire ecosystem of open-source language models effectively relies on a single platform.
By Jacob Nielsen, Danial Namazifard, Lukas Galke Poech, Peter Schneider-Kamp