AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

18,070 stories · RSS feed

arXiv Machine Learning
Jul 22

Estimating near-verbatim extraction risk in language models with decoding-constrained beam search

arXiv:2603. 24917v2 Announce Type: replace-cross Abstract: Recent work shows that standard greedy-decoding extraction methods for quantifying memorization in LLMs miss how extraction risk varies across sequences.

By A. Feder Cooper, Mark A. Lemley, Christopher De Sa, Lea Duesterwald, Allison Casasola, Jamie Hayes, Katherine Lee, Daniel E. Ho, Percy Liang
arXiv Machine Learning
Jul 22

Signed Rectified Flow: Negativity-Controlled Generation

arXiv:2607. 18516v1 Announce Type: new Abstract: We introduce Signed Rectified Flow (Signed RF), a generalization of Rectified Flow that targets the signed measure $\pi^{sign} = (1+\alpha)\pi^+ - \alpha\pi^-$, where $\alpha>0$, $\pi^+$ is the distribution to promote, and $\pi^-$ is the distribution to suppress.

By Runlong Liao, Baiyu Su, Lizhang Chen, Qiang Liu
arXiv AI
Jul 22

SAAG: Structured Agent Assessment and Grounding

arXiv:2607. 18245v1 Announce Type: new Abstract: Exact-match evaluation of agent-calling obscures qualitatively different failure modes: a model may select the right function yet hallucinate argument values, or satisfy a schema while choosing a agent for the wrong reason.

By Ritvik Garimella, Vedant Khandelwal, Anvi Kohli, Amit Sheth
arXiv AI
Jul 22

FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification

arXiv:2604. 03245v2 Announce Type: replace-cross Abstract: The remarkable reasoning and code generation capabilities of large language models (LLMs) have recently motivated increasing interest in automating formal verification (FV), a process that ensures hardware correctness through mathematically precise assertions but remains highly labor-intensive, particularly through the translation of natural language into SystemVerilog Assertions (NL-to-SVA).

By Lily Jiaxin Wan, Chia-Tung Ho, Yunsheng Bai, Cunxi Yu, Ghaith Bany Hamad, Deming Chen, Haoxing Ren