arXiv Machine Learning By Shai Shalev-Shwartz, Amnon Shashua

Alignment via Generalization

Read the original on arXiv Machine Learning →

The paper introduces Aligned‑PAC learning, a framework that reframes the alignment problem as a question of generalization: whether success on training requirements extends to unseen ones. It proposes representing intentions through multiple specific requirements, provides theoretical results on independent evaluation and the amount of evaluation needed, and develops an optimization method that combines learned models to satisfy these requirements under certain assumptions. The authors acknowledge that selecting appropriate requirements and ensuring generalization to new ones remain open challenges, positioning their work as a foundation for precise, learning‑theoretic study of alignment.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Oct 9

Predicting Alignment Generalization with Value Representations

The paper introduces the task of predicting alignment generalization, which measures how fine‑tuning a language model to follow a specific value affects its behavior across a broad set of other values. Through a large‑scale study of 66 values, the authors show that representations derived from model activations when applying values in context outperform textual description‑based methods, achieving a correlation of 0.45 versus 0.05. They further demonstrate that these representations can predict downstream robustness and help construct a taxonomy of LLM values based on empirical generalization dynamics.

By Andy Liu, Mehar Bhatia, Karolina Stanczak, Mona Diab, Vered Shwartz, Daniel Fried
OpenAI Blog
Aug 24, 2022

Our approach to alignment research

We are improving our AI systems’ ability to learn from human feedback and to assist humans at evaluating AI. Our goal is to build a sufficiently aligned AI system that can help us solve all other alignment problems.