Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks
Read the original on arXiv Machine Learning →The paper investigates whether Prior‑Data Fitted Networks (PFNs) encode spectral information about their input context. Across seven PFNs—including pretrained tabular foundation models and a decision‑tree prior model—a linear probe on the residual stream recovers the context’s frequency spectrum with high accuracy (R² ≥ 0.95), driven by a single principal direction. Activation and subspace patching show that the network exploits this spectral structure in a low‑dimensional subspace, and a Filter Bank Decoder can transform frozen PFN representations into an explicit stationary kernel that enables competitive Gaussian process regression at far lower cost.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.