Hugging Face Trending Papers

Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training

Read the original on Hugging Face Trending Papers →

The paper investigates how per-domain data composition during the mid‑training phase (between pre‑training and alignment) affects model performance. Experiments with Qwen3‑8B‑Base across five KOR‑Bench domains show that a moderate coverage band (10%‑40%) yields the best performance for each domain, and that alignment passes cannot fully close the gaps created by suboptimal mid‑training allocations. Additionally, zero coverage during mid‑training severely degrades accuracy, while a carefully tuned allocation can provide the largest overall pipeline improvement.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv AI
Sep 10

Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training

The study investigates how the composition of data during the mid‑training phase of language models affects performance across multiple domains. Experiments with Qwen3‑8B‑Base on five distinct KOR‑Bench domains show that moderate coverage (10%‑40%) yields the best per‑domain results, and that alignment passes cannot fully close the performance gaps created by mid‑training data choices. Additionally, zero coverage in mid‑training severely degrades accuracy, while a carefully tuned allocation can provide the largest overall pipeline improvement.

By Yunpeng Xu, Kun Zheng
arXiv AI
3d ago

How many LLM-labelled rows does a decision encoder need, and how wrong can the labels be?

The study investigates how many large‑language‑model (LLM)‑generated training rows a decision encoder, specifically Laya (ModernBERT‑large), requires to achieve high accuracy on a choice‑making task. Using 19 fine‑tunes on nested subsets of a 2,818‑row pool, accuracy improves from 69.4 % before fine‑tuning to 86.9 % with all rows, with diminishing returns as more data is added. The experiment also examines the impact of label noise, finding that 10 % symmetric noise has negligible effect, while 20 % and 40 % noise degrade performance by 3.4 and 8.7 points respectively, and that a single run with 10 % confusable‑pair noise reduces confidence. whyItMatters":"The results quantify the data efficiency and robustness of Laya to noisy LLM‑generated labels, informing how much synthetic training data is needed for reliable decision‑encoding models."

By Mohammed Nayeem Khan, Aliyas Ahmad Misri
arXiv AI
Oct 16

Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage

The paper argues that post‑training pruning of large language models should use a calibration set that covers multiple capability domains rather than a single source. A balanced four‑way mixture of sources consistently outperforms any single source across sparsity levels, model scales, and held‑out benchmarks. When aligned corpora are unavailable, the IGSP method approximates this mixture using only the base model and an evaluation taxonomy, achieving significant gains over prior self‑generation approaches.

By Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu
arXiv Machine Learning
Nov 5

The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention

The study empirically investigates how length‑normalized Direct Preference Optimization (DPO) with a Structured Fine‑Tuning (SFT) retention term (RPO objective) affects downstream capabilities of a Qwen3.5‑4B model fine‑tuned with sparse LoRA on structured financial‑filing tasks. Results show that the alignment tax is graded and task‑selective: extraction performance is largely preserved, JSON extraction degrades modestly, and router‑based tasks suffer severe loss, with most degradation attributed to suppression of rejected completions. The analysis uses 8 seeds, artifact‑aware scoring, and TOST equivalence testing to quantify these effects across different reward margins and precision settings.

By Ye Yuan