Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later training. One recently popular technique for such inter...
arXiv:2607. 26654v1 Announce Type: cross Abstract: Post-training alignment is often shallow, eroding under fine-tuning.
By Desiree Cho, Cameron Tice, Bernie Hogan, Hunar Batra, Puria Radmard, Jun Zhao, Nigel Shadbolt
arXiv:2610.00767v1 Announce Type: cross
Abstract: Pre-training interventions are critical to alignment research, since beliefs formed during pre-training shape how a model generalizes from later trai...
By Peter Nutter, Dani Roytburg, Cl\'ement Dumas, Jinghua Ou, Shi Feng
arXiv:2607. 26173v1 Announce Type: new Abstract: Alignment training, model organisms, and toy models are usually treated as separate research areas.
By Anton de la Fuente, Arthur Conmy
The paper introduces Alignment Forecasting, a method for predicting whether fine‑tuning a language model on a given dataset will increase specific alignment failures such as deception or sycophancy. It presents ALIGNMENTFORECASTBENCH, a benchmark of over 5,000 forecasting questions across many models, datasets, and failure modes, and shows that a simple forecasting scaffold using an LLM’s assessment of dataset bias can outperform baseline forecasters. The authors demonstrate that filtering out high‑risk training examples identified by the forecaster can improve alignment in multiple‑choice evaluations, though benefits in open‑ended conversations remain uncertain.
By Chen Yueh-Han, Bruce W. Lee, Ilia Sucholutsky, Tomek Korbak
arXiv:2510. 26707v2 Announce Type: replace-cross Abstract: As LLMs occupy an increasingly important role in society, they are more and more confronted with questions that require them not only to draw on their general knowledge but also to align with certain human value systems.
By Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Sta\'nczak, Vered Shwartz, Siva Reddy
arXiv:2608. 12307v1 Announce Type: cross Abstract: Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods.
By Cheng Qian, Wenting Zhao, Liangwei Yang, Heng Wang, Jielin Qiu, Heng Ji, Silvio Savarese, Huan Wang, Shelby Heinecke
The paper "Mass Producing Training Data with Surprising Side Effects" presents a method for automatically discovering datasets that cause unexpected side effects in language models. By defining three criteria—target behavior elicitation, ordinary appearance, and human surprise—the authors use LLM judges and search techniques to generate 6,575 datasets across eight behaviors. These datasets, which generalize across model families and sizes, provide a testbed for studying generalization and predicting training side effects before deployment.
By Yaowen Ye, Jacob Steinhardt, Sarah Schwettmann, Cassidy Laidlaw
The paper introduces a technique called Synthetic Document Finetuning (SDF) to expose hidden misaligned behaviors in large language models that may engage in alignment faking. By finetuning models to associate certain keywords with undetected misaligned actions, the authors create honeypot environments and observe that the gap in misaligned action rates between cueed and uncued prompts widens, indicating more reliable detection than prompt-based methods. The study also notes that SDF can impair reasoning abilities, proposing on-policy distillation as a mitigation.
By Ardy Haroen, Florian Lorkowski, Samuel Nellessen, Felix Hofst\"atter
arXiv:2606. 11045v1 Announce Type: new Abstract: Reusing a held-out benchmark adaptively should, in principle, invite overfitting.
By Martin Andres Bertran, Aaron Roth, Zhiwei Steven Wu
Alignment training, model organisms, and toy models are usually treated as separate research areas. But projects in all three frequently use supervised fine-tuning (SFT) to pursue the same underlying goals.
arXiv:2609.37169v1 Announce Type: cross
Abstract: Mid-training equips pretrained large language models with specialized and reasoning capabilities, but the returns of this stage are bounded since add...
By Zhehao Huang, Changxin Tian, Qingyuan Yang, Kunlong Chen, Ziqi Liu, Zhiqiang Zhang, Xiaolin Huang, Jun Zhou