The paper investigates how synthetic pretraining priors used in tabular foundation models (TFMs) influence downstream performance. By reconstructing the synthetic data generators of four TFMs and comparing their generated tasks to two popular tabular benchmarks using structural descriptors, the authors measure structural coverage and normalized density. They find that some generators provide broader and denser support for benchmark tasks, and that stronger synthetic-to-benchmark support generally correlates with better model performance.
By He Zhao, Ryan Thompson, Daniel M. Steinberg, Ashfaqur Rahman, Edwin V. Bonilla, Cheng Soon Ong
The paper proposes a generation‑provenance substrate for synthetic speech research objects, binding source specifications, generated content, waveform, target, fact requirements, quality signals, review lineage, and an immutable manifest identity. It audits this substrate in a private Japanese care‑handoff pipeline, documenting 113 assets and 1.552 hours of synthetic speech with linked audio, transcripts, notes, and fact checklists, while noting selective human evidence and source‑specific gaps. The authors argue that provenance is necessary but not sufficient for behavior attribution, requiring additional frozen training runs and intervention evidence, and they provide a compact provenance contract, audit protocol, and a bounded case study.
"whyItMatters":"The study highlights the need for detailed provenance records to enable reliable auditing and attribution of synthetic data behavior, underscoring limitations in current practices and offering a structured framework for future research."
By Sidi Chang, Peiying Zhu
Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later training. One recently popular technique for such inter...
arXiv:2610.00767v1 Announce Type: cross
Abstract: Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later trai...
By Peter Nutter, Dani Roytburg, Cl\'ement Dumas, Jinghua Ou, Shi Feng
arXiv:2606. 15390v1 Announce Type: cross Abstract: LLM agents can improve without weight updates by accumulating natural-language skills from experience, but current systems entrust every decision about which skills to keep and how to apply them to LLM judgment alone.
By Yixuan Wang, Yiyang Zhou, Yiming Liang, Congyu Zhang, Fuxiao Liu, Jiawei Zhou, Huaxiu Yao
The study investigates whether provenance information can reliably identify the source of synthetic text and whether this identification improves the selection of training data. Using financial‑risk text, the authors achieve 98.7% accuracy in attributing original generated passages, but accuracy drops to 53.1% after paraphrasing and 29.0% after style rewriting. They compare two selection strategies—one based on source provenance and another on a reference model score—across three rounds of generation and retraining, finding that the two methods choose different examples but do not produce a consistent difference in model degradation. The results suggest that source attribution and useful data selection are distinct challenges, and neither provenance nor the tested proxy suffices to guarantee stable recursive training behavior.
By Joss Armstrong
The paper investigates how training data attribution (TDA) can be used to influence large language models (LLMs). It compares two methods—reweighting and influence-guided response rewriting—on examples selected by influence functions. Rewriting, which replaces responses while keeping instructions fixed, yields stronger, more persistent, and bidirectional behavioral changes than reweighting, suggesting that the intervention value of influential samples is better realized through rewriting.
By Yuzhang Luo, Chenpeng Wang, Jianhui Chen, Liangming Pan
LUMOS is a diagnostic framework that tracks how knowledge in large language models (LLMs) originates from training data and manifests in outputs, using the fully transparent OLMo 2 corpus. The study finds that while models encode rare facts with high separability (84%), they often fail to express them behaviorally (54%), and self‑reflection accuracy drops sharply on unseen content. These results show that incorporating the training‑data axis into evaluation turns speculative claims into verifiable evidence, suggesting it should become a standard part of LLM knowledge assessment.
By Seoyeon Ye, Gayoung Kim, Jiyoung Hong, Sookyung Kim, Hyunsoo Cho
The paper investigates why certain synthetic relational datasets lead to better relational foundation models. By training four Relational Transformer checkpoints on data from four different generators, the authors trace a measurable property of the data—specifically the predictive necessity of cross‑table information—to the emergence of relational computation in the models. They find that the RelDiff generator yields the largest predictive gain from foreign‑key‑linked parents, and its model uniquely responds to foreign‑key interventions, a dependence that persists across random initializations and grows with corrupted links. Disrupting this mechanism during downstream inference eliminates RelDiff’s advantage on relational tasks while leaving structure‑insensitive models largely unchanged, thereby linking synthetic data properties to learned mechanisms and downstream behavior.
By Shivam Dubey, Mohamed Bouadi, Nassim Bouarour, Varun Kulkarni, Aditya Tanna, Vinay Kumar Sankarapu
arXiv:2607. 29484v1 Announce Type: cross Abstract: Interventional data is widely regarded as the gold standard for teaching models causal reasoning.
By Xining Xun
arXiv:2608. 05235v1 Announce Type: cross Abstract: Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings and retain the resulting trajectories to guide later decisions.
By Zijie Zhuang, Changxin Lao, Pengbo Xu, Hanwen Xu, Ruochen Yang, Yingzhi He, Peng Zhang, Jiangxia Cao, Yusheng Huang, Guohong Mu, Jian Liang, Ruiming Tang, Shuang Yang, Zhaojie Liu, Wenwu Ou, Kun Gai
arXiv:2607. 23804v1 Announce Type: cross Abstract: Context attribution methods for large language models (LLMs) identify which input context contributes to the model response.
By Quoc-Huy Trinh, Lin Zhu, Sebastian Szyller