arXiv Machine Learning
Jul 1
A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
arXiv:2602. 20062v2 Announce Type: replace Abstract: Pretraining and fine-tuning are central stages in modern machine learning systems.
From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging
arXiv:2506. 14126v2 Announce Type: replace-cross Abstract: Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets.
Related stories
arXiv AI
Aug 14
CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation
arXiv:2608. 12842v1 Announce Type: new Abstract: Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining.
arXiv Machine Learning
Aug 6
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
arXiv:2608. 05000v1 Announce Type: cross Abstract: Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining.
arXiv AI
Sep 10
DUA-D2C: Dynamic Uncertainty Aware Method for Overfitting Remediation in Deep Learning
The paper introduces DUA-D2C, a Dynamic Uncertainty-Aware Divide2Conquer method that improves overfitting remediation in deep learning. It refines the traditional Divide2Conquer approach by dynamically weighting subset models based on a composite score of accuracy and normalized prediction entropy, allowing the central model to learn more from generalizable and confident edge models. The authors provide theoretical justification, show reduced model variance, and demonstrate significant generalization gains across image, audio, and text benchmarks, even when combined with standard regularizers like Dropout.
arXiv Machine Learning
Jun 4
Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform
arXiv:2603. 07523v3 Announce Type: replace Abstract: Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales.
arXiv Machine Learning
Jun 16
Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
arXiv:2601. 18699v2 Announce Type: replace Abstract: Sequential fine-tuning of Large Language Models (LLMs) adaptation to target tasks often triggers catastrophic forgetting, where the acquisition of novel target skills degrades ancestral capabilities.
arXiv Machine Learning
Jul 15
Inference-Time Machine Unlearning via Gated Activation Redirection
arXiv:2605. 12765v3 Announce Type: replace Abstract: Large Language Models memorize vast amounts of training data, raising concerns regarding privacy, copyright infringement, and safety.
arXiv Machine Learning
Jun 18
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
arXiv:2606. 18627v1 Announce Type: new Abstract: Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model.
arXiv AI
4d ago
Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories
arXiv:2609.37169v1 Announce Type: cross Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since add...
arXiv Machine Learning
Sep 11
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
The paper investigates how data repetition affects Mixture-of-Experts (MoE) language models compared to dense Transformers. Across models from 80 M to 1 B active parameters, MoEs degrade more quickly as data is repeated, with performance dropping significantly beyond 4× repetition and overtaking dense models only when strong regularization is applied. The study also identifies routing stabilization and expert specialization as key factors in MoE overfitting, and explores regularization techniques that can partially mitigate this issue.
arXiv Machine Learning
4d ago
UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?
arXiv:2609.37076v1 Announce Type: new Abstract: Large language models trained on vast corpora inherently risk memorizing harmful content that may later re-emerge in their outputs. To mitigate this is...
arXiv Machine Learning
Sep 22
Merge++: Universal Merge Refinement Through Data-Free Checkpoint Inversion
Merge++ is a post‑hoc refinement technique for model merging that synthesizes task‑representative images by inverting expert checkpoints and then distills expert knowledge into a merged model. It operates without any additional data beyond the checkpoints and can be applied universally across existing weight‑space merging algorithms. Experiments show consistent improvements, with average gains of +2 to +8 points and up to +25.9 on specific configurations.