← Back to all news
arXiv Machine Learning

Is spectral predictability enough? Testing {\Omega}-matching for synthetic pretraining under distribution shift

The paper investigates whether matching the spectral predictability Ω of synthetic pretraining data to a target dataset is sufficient for effective transfer learning under distribution shift. Experiments across LSTM, PatchTST, and TCN architectures show that Ω-matched pretraining can outperform an oracle baseline in synthetic-to-synthetic settings, but on the real-world ETTm1 benchmark it offers only marginal gains or even degrades performance, with Ω providing little guidance on beneficial source selection. The authors conclude that while Ω matching can help in controlled scenarios, it is not a reliable predictor of transfer success and other signal properties must be considered.

Related stories

arXiv AI
Sep 16

Schema-Adaptive Action-Conditioned JEPA for Cross-Machine CNC Transfer under Partial Sensor Overlap

The paper introduces a schema‑adaptive action‑conditioned Joint‑Embedding Predictive Architecture (SAAC‑JEPA) for cross‑machine CNC transfer when only a subset of sensors overlap between source and target machines. Experiments show that pretraining does not improve source‑only forecasting, but a carefully selected action‑conditioned JEPA model achieves a zero‑shot RMSE of 0.546 on the target, outperforming persistence but falling short of certain baseline models. Ablation studies reveal that adding RevIN improves RMSE but harms calibration, and limited post‑lock adaptation can further reduce error.

By Ayoub Louaye Bouaziz, Matthieu Ostertag, Anton Demasles
ragbenchmarks
More like this →
arXiv Machine Learning
Oct 22

Minimax-Optimal Predictive Flow Matching under Distribution Drift

The paper introduces DyProd-FM, a method for predictive flow matching that refits flow matching models on dyadic historical prefixes and selects among them using a sleeping Prod master. It achieves dynamic regret bounds of “~O(min{N_ω+1,1+√{TΩ}}) and ~O(min{N_μ+1,1+T^{1/3}V_μ^{2/3}}) in relative velocity risk variation Ω and target‑law variation V_μ, respectively, without requiring a drift budget. These bounds are shown to be optimal up to logarithmic factors for fixed, possibly nonconvex, neural classes.

By Aadithya Srikanth, Vaneet Aggarwal
diffusion
More like this →
arXiv AI
Oct 27

Self-Supervised Pretraining for PHM: What Transfers, What Does Not, and What Federating It Costs

The study evaluates self‑supervised pretraining for Prognostics and Health Management (PHM) across 47 public datasets, using a compact Transformer encoder trained with masked patch prediction. Transfer performance is highly task‑dependent: it dramatically improves fault‑classification on CWRU and HSG18 but offers no reliable benefit for remaining‑useful‑life estimation on CMAPSS. The authors also compare centralized versus simulated federated pretraining, finding that centralized training generally outperforms federated training under matched budgets, and that the intrinsic SSL loss does not reliably predict downstream performance.

By Jos\'e Sanz-Dur\'an, David Sol\'is-Mart\'in
llms
More like this →
arXiv AI
Aug 25

Improving Few-Step Language Flows with Untied Self-Conditioning

The paper introduces Untied Self-Conditioning, a sampler that corrects a train–inference mismatch in flow‑matching language models. By dampening redundant directions in the self‑conditioning input and approximating a step‑average prediction from history, the method improves generation quality without retraining. On LangFlow and ELF‑B datasets, it dramatically lowers perplexity and is preferred in the majority of pairwise comparisons.

By Bocheng Li, Linli Xu
llms
More like this →