Hugging Face Trending Papers

Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training.

arXiv Machine Learning

1d ago

Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining

The study investigates how layer‑wise intervention responses in language models change over the course of pretraining, using single‑block identity bypass across multiple checkpoints and model‑domain combinations. It finds that while depth ordering of responses persists, their magnitudes shift, with nearby checkpoints showing stronger rank correspondence than distant ones and large changes occurring at positions that recur across samples and transfer across evaluation domains. Controlled experiments reveal that these longitudinal changes cannot be explained by a single downstream sensitivity and depend on perturbation strength and direction, indicating that layer sensitivity is structured but dynamic.

By Shengye Tao, Yinzhu Cheng, Haihua Xie

llms

arXiv AI

4d ago

Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories

arXiv:2609.37169v1 Announce Type: cross Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since add...

By Zhehao Huang, Changxin Tian, Qingyuan Yang, Kunlong Chen, Ziqi Liu, Zhiqiang Zhang, Xiaolin Huang, Jun Zhou

llms safety

arXiv AI

Sep 10

Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

arXiv:2609.08966v1 Announce Type: new Abstract: Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the...

By Sohir Maskey, Philipp Scholl, Jonas Knupp, Pit Neitemeier, Sascha Wirges

fine-tuning benchmarks

arXiv AI

Jun 18

From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging

arXiv:2506. 14126v2 Announce Type: replace-cross Abstract: Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets.

By Stefan Horoi, Guy Wolf, Eugene Belilovsky, Gintare Karolina Dziugaite

fine-tuning

arXiv AI

Sep 1

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

The study investigates how post‑training of large autoregressive language models (ARMs) into masked diffusion models (MDMs) affects their internal computation. Across two 7B ARM‑MDM families and four diagnostic tasks, the authors find that MDMs retain much of the ARM’s high‑attribution pathways on prefix‑dominant tasks, but reorganize computation toward earlier layers on globally constrained tasks. Component‑level probes reveal that ARMs depend on sharply specialized components, whereas MDMs show weaker specialization and more diffuse output‑space alignment.

By Injin Kong, Hyoungjoon Lee, Yohan Jo

llms diffusion safety

arXiv Machine Learning

Jul 13

Complexity-Guided Component-wise Initialization for Language Model Pretraining

arXiv:2607. 09204v1 Announce Type: cross Abstract: Pretrained language models often exhibit structured weight spectra, suggesting that training may repeatedly produce similar layerwise and component-wise organization.

By Konstantin Garbers, Nicholas Oh

llms nlp

arXiv AI

2d ago

Pre-training interventions, ex post facto: Grafting model beliefs across checkpoints

arXiv:2610.00767v1 Announce Type: cross Abstract: Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later trai...

By Peter Nutter, Dani Roytburg, Cl\'ement Dumas, Jinghua Ou, Shi Feng

fine-tuning safety

arXiv AI

Jun 18

DRIFT: Refining Instruction Data via On-Policy Data Attribution

arXiv:2606. 18307v1 Announce Type: cross Abstract: Optimizing the training data distribution for Supervised Fine-Tuning (SFT) dictates the capability of Large Language Models (LLMs).

By Zefan Wang, Lincheng Li, Tianyu Yu, Yuan Yao

llms fine-tuning safety

arXiv Machine Learning

Aug 18

MoRFI: Monotonic Sparse Autoencoder Feature Identification

arXiv:2604. 26866v2 Announce Type: replace-cross Abstract: Large language models (LLMs) acquire most of their factual knowledge during the pre-training stage, through next token prediction.

By Dimitris Dimakopoulos, Shay B. Cohen, Ioannis Konstas

llms fine-tuning safety

arXiv AI

4d ago

From Checkpoint Variation to Selection Gains in Supervised Fine-Tuning

arXiv:2609.36569v1 Announce Type: cross Abstract: Checkpoint selection is a routine decision in supervised fine-tuning (SFT): training produces multiple checkpoints, but only one is retained. Yet fix...

By Yupeng Chang, Wenxuan Zhang, Yuan Wu

fine-tuning

arXiv Machine Learning

Jun 4

STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations

arXiv:2606. 05165v1 Announce Type: new Abstract: Training Data Attribution (TDA) seeks to trace a model's predictions back to its training data.

By Rishit Dagli, Abir Harrasse, Luke Zhang, Florent Draye, Amirali Abdullah, Bernhard Sch\"olkopf, Zhijing Jin

llms benchmarks

arXiv Machine Learning

Jul 1

A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup

arXiv:2605. 11554v2 Announce Type: replace Abstract: Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another, but such explanations require the proxy to track the structure that matters for the downstream task.

By Hongmin Li

Related stories