The study evaluates self‑supervised pretraining for Prognostics and Health Management (PHM) across 47 public datasets, using a compact Transformer encoder trained with masked patch prediction. Transfer performance is highly task‑dependent: it dramatically improves fault‑classification on CWRU and HSG18 but offers no reliable benefit for remaining‑useful‑life estimation on CMAPSS. The authors also compare centralized versus simulated federated pretraining, finding that centralized training generally outperforms federated training under matched budgets, and that the intrinsic SSL loss does not reliably predict downstream performance.
Self-supervised pretraining has produced capable time-series foundation models, but its value for Prognostics and Health Management (PHM) has not been measured under leakage-controlled conditions. We...
arXiv:2607. 16681v1 Announce Type: new Abstract: Early sepsis prediction from electronic health records is challenged by irregular sampling, high missingness, and class imbalance.
arXiv:2607. 00958v1 Announce Type: new Abstract: Time series are central to modern data mining applications, from industrial telemetry and server metrics to finance and physiology, yet time-series self-supervised learning often depends on view and augmentation choices that encode domain-specific invariances.
arXiv:2605. 10840v3 Announce Type: replace-cross Abstract: We present Clin-JEPA, a multi-phase co-training framework for joint-embedding predictive (JEPA) pretraining on EHR patient trajectories.
The paper investigates how synthetic tasks influence language‑model pretraining by distinguishing diagnostic, teachable, and transferable signals. In a controlled setting where 70% of the data is general Python and 30% is a mixture of curated synthetic tasks, literature‑derived probes, and OpenCodeInstruct, 14 of 27 tasks show teachability, with curated tasks outperforming literature probes. Curated synthetic data provides conditional value: it boosts downstream performance when mixed with OpenCodeInstruct but is less effective when used alone, and a loss‑based scheduler can misalign short‑term loss reduction with long‑term transfer.
arXiv:2608. 06122v1 Announce Type: cross Abstract: Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series.
The paper demonstrates that greedy decoding from large language models is not precision‑invariant: the same model, prompt, and decoding algorithm can produce different outputs when run in BF16 versus FP16 on identical hardware. Across six models (1.1B–7B parameters, four families, and 12B) and three benchmarks, 49–100 % of prompts diverge, with a single token flip often cascading into trajectory‑level divergence. The authors develop an empirical error‑propagation analysis that identifies the top‑two logit margin at the LM head as the key factor, and they propose a low‑overhead intervention—selective FP32 LM head recomputation—that improves exact agreement by 22–36 percentage points with less than 4 % latency overhead. "whyItMatters":"The findings reveal that precision choices can fundamentally alter model outputs, challenging the assumption of deterministic greedy decoding and highlighting the need for precision‑aware inference strategies."
The paper introduces a schema‑adaptive action‑conditioned Joint‑Embedding Predictive Architecture (SAAC‑JEPA) for cross‑machine CNC transfer when only a subset of sensors overlap between source and target machines. Experiments show that pretraining does not improve source‑only forecasting, but a carefully selected action‑conditioned JEPA model achieves a zero‑shot RMSE of 0.546 on the target, outperforming persistence but falling short of certain baseline models. Ablation studies reveal that adding RevIN improves RMSE but harms calibration, and limited post‑lock adaptation can further reduce error.
arXiv:2606. 18114v1 Announce Type: cross Abstract: State Space Models (SSMs) such as Mamba-2 offer linear-time inference but their memory footprint limits edge deployment.
arXiv:2609.37642v1 Announce Type: new Abstract: Self-supervised pretraining reshaped prediction in language and vision, and brain foundation models (BFMs) inherited its promise. Representations learn...
arXiv:2608.24593v1 Announce Type: new Abstract: Adaptive optimizers retain gradient history in moment variables, allowing a local change in loss weighting to alter later updates. We examine whether t...
arXiv:2610.17573v1 Announce Type: cross Abstract: Self-supervised pretraining has expanded the pool of 3D medical foundation models, but published evaluations use different datasets, endpoints and fi...