AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,896 stories · RSS feed

arXiv AI
Aug 18

Rethinking Generalization in Reasoning SFT: A Conditional Analysis on Optimization, Data, and Model Capability

arXiv:2604. 06628v2 Announce Type: replace Abstract: A prevailing narrative in LLM post-training holds that supervised finetuning (SFT) memorizes while reinforcement learning (RL) generalizes.

By Qihan Ren, Peng Wang, Ruikun Cai, Shuai Shao, Dadi Guo, Yuejin Xie, Yafu Li, Quanshi Zhang, Xia Hu, Jing Shao, Dongrui Liu
arXiv AI
Aug 18

Position: AI Governance Needs ISO-like Interoperability Protocols, Not Just Laws

The paper argues that AI governance should rely on ISO-like interoperability protocols rather than solely on jurisdiction-specific laws. It proposes standardized AI nutrition labels that include metrics for bias, energy usage, and data provenance to enable machine‑readable risk communication across borders. These protocols aim to reduce regulatory fragmentation, lower barriers for SMEs, and build public trust while allowing modular evolution with technology.

By Azmine Toushik Wasi, Mst Rafia Islam, Mahfuz Ahmed Anik, Taki Hasan Rafi, Md Manjurul Ahsan, Dong-Kyu Chae
arXiv AI
Aug 18

OGX: An Open-Source, Vendor-Neutral Generative AI Application Server

OGX (Open GenAI Stack) is an open‑source AI application server and Python library that implements the APIs of major frontier labs such as OpenAI, Anthropic, and Google, allowing developers to build agentic AI applications against a single API surface. It supports a wide range of inference providers, vector store backends, and safety backends, and includes a Kubernetes Operator for production deployment. OGX is used as the self‑hosted, model‑agnostic backend for AI‑powered developer tools like Claude Code, Codex CLI, OpenCode, and OpenHands, and has garnered significant community engagement with over 8,400 GitHub stars, 242 contributors, and 4,000 commits.

By Francisco Javier Arceo, S\'ebastien Han, Matthew Farrellee, Charlie Doern, Yuan Tang, Derek Higgins, Varsha Prasad Narsing, Gordon Sim, Sumanth Kamenani, Ben Browning, Raghotham Murthy
arXiv AI
Aug 18

MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment

arXiv:2608. 14828v1 Announce Type: new Abstract: Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help.

By Tony Tu, Sayan Chakraborty, Ruomeng Xu, Tony Qin, Austin Tian