The study investigates how the composition of data during the mid‑training phase of language models affects performance across multiple domains. Experiments with Qwen3‑8B‑Base on five distinct KOR‑Bench domains show that moderate coverage (10%‑40%) yields the best per‑domain results, and that alignment passes cannot fully close the performance gaps created by mid‑training data choices. Additionally, zero coverage in mid‑training severely degrades accuracy, while a carefully tuned allocation can provide the largest overall pipeline improvement.
By Yunpeng Xu, Kun Zheng
The study investigates how many large‑language‑model (LLM)‑generated training rows a decision encoder, specifically Laya (ModernBERT‑large), requires to achieve high accuracy on a choice‑making task. Using 19 fine‑tunes on nested subsets of a 2,818‑row pool, accuracy improves from 69.4 % before fine‑tuning to 86.9 % with all rows, with diminishing returns as more data is added. The experiment also examines the impact of label noise, finding that 10 % symmetric noise has negligible effect, while 20 % and 40 % noise degrade performance by 3.4 and 8.7 points respectively, and that a single run with 10 % confusable‑pair noise reduces confidence.
whyItMatters":"The results quantify the data efficiency and robustness of Laya to noisy LLM‑generated labels, informing how much synthetic training data is needed for reliable decision‑encoding models."
By Mohammed Nayeem Khan, Aliyas Ahmad Misri
The paper argues that post‑training pruning of large language models should use a calibration set that covers multiple capability domains rather than a single source. A balanced four‑way mixture of sources consistently outperforms any single source across sparsity levels, model scales, and held‑out benchmarks. When aligned corpora are unavailable, the IGSP method approximates this mixture using only the base model and an evaluation taxonomy, achieving significant gains over prior self‑generation approaches.
By Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu
The study empirically investigates how length‑normalized Direct Preference Optimization (DPO) with a Structured Fine‑Tuning (SFT) retention term (RPO objective) affects downstream capabilities of a Qwen3.5‑4B model fine‑tuned with sparse LoRA on structured financial‑filing tasks. Results show that the alignment tax is graded and task‑selective: extraction performance is largely preserved, JSON extraction degrades modestly, and router‑based tasks suffer severe loss, with most degradation attributed to suppression of rejected completions. The analysis uses 8 seeds, artifact‑aware scoring, and TOST equivalence testing to quantify these effects across different reward margins and precision settings.
By Ye Yuan
arXiv:2610.17093v1 Announce Type: new
Abstract: A model diff credits post-training with whatever separates a base checkpoint from one post-trained descendant, presuming a rerun would land close by. O...
By Xupeng Chen, Chenqian Le, Jiayi Cheng, Runhao Li, Michael Yue
arXiv:2607. 17136v1 Announce Type: cross Abstract: Agentic computer-use RL is reported in single runs, and those numbers mislead.
By Barada Sahu (Cabal AI), Shivesh Pandey (Para AI)
arXiv:2609.36426v1 Announce Type: cross
Abstract: A detector pretrained on a broad corpus is fine-tuned on a narrow domain, its in-domain accuracy improves, and it ships. We ask what happens meanwhil...
By Trung Minh Bui, Jongsul Moon, YoungOuk Kim, Jung-Hoon Hwang, Dongin Shin
The paper evaluates cost‑aware abstention for routing IT support tickets, comparing four non‑LLM classifiers and four large language models on a 47,837‑ticket, eight‑class dataset. Using a cost model (c_ok=0, c_triage=1, c_wrong=5), a TF‑IDF logistic regression model with a 0.60 threshold reduced test cost from 0.763 to 0.472 and lowered wrong routes from 15.3% to 3.5%. The study also shows that calibration‑selected policies outperform always‑route and always‑defer strategies in most settings and introduces new metrics such as correctness AUROC and tie‑aware risk‑coverage area.
By Sachin Gupta
The study measured the impact of a single training example on a GPT‑2 model by running 24 counterfactual experiments. 32 models were trained from scratch on OpenWebText, and at a specific training step a single batch row was replaced with a 194‑token passage under three conditions (fluent prose, fabricated subject, random characters) or left unchanged. Results showed that the passage was learned from one exposure and decayed, with measurable differences in cross‑entropy up to 50 steps after injection but no lasting effect at the final step.
By Zachary Speck, Asa Shepard
The paper introduces a selection‑safe conformal early‑exit method that calibrates a single split‑conformal quantile using a terminal‑path score derived from prefix maxima. This approach guarantees coverage for any data‑dependent stopping policy that preserves terminal‑set containment, without requiring an α/L correction. Empirical results on eight encoder backbones and CIFAR‑100 show that pathwise selection achieves higher macro coverage (0.901 vs. 0.814) with smaller set sizes and lower latency fractions compared to naive layer‑wise selection, and remains robust to deployment‑time budget changes.
By Adria Binte Habib, Ibne Farabi Shihab
The study evaluates how to best split tasks among large‑language‑model agents for cross‑border VAT determination, comparing one broad agent to configurations ranging from one to five narrow agents. Across 4,400 runs—including token‑matched and failure‑injection scenarios—the intermediate configurations achieved the highest accuracy but did not surpass the fine‑endpoint benchmark, leaving the optimal decomposition hypothesis unconfirmed. The pilot provides a preregistered heuristic for right‑sizing decomposition, along with an oracle, dataset, and analysis pipeline.
By Pedro Santos
arXiv:2608. 13087v1 Announce Type: cross Abstract: Neural combinatorial optimization (NCO) solvers report the best of many sampled solutions per instance, and the sample count is, by convention, identical for every instance.
By Jinhyung Bae