AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Machine Learning
Sep 21

K2-V2: A 360-Open, Reasoning-Enhanced LLM

K2-V2 is a fully open, 360‑open large language model built from scratch, designed to serve as a superior base for reasoning adaptation while also supporting conversation and knowledge retrieval. It competes with leading open‑weight models in its size class, outperforming Qwen2.5‑72B and approaching Qwen3‑235B, and incorporates domain knowledge, reasoning, long‑context handling, and tool use throughout training. The authors release the complete training history, data composition, model weights, and LLM360 artifacts to enable community use and continuous training.

By K2 Team, Zhengzhong Liu, Liping Tang, Linghao Jin, Haonan Li, Nikhil Ranjan, Desai Fan, Shaurya Rohatgi, Richard Fan, Omkar Pangarkar, Huijuan Wang, Zhoujun Cheng, Suqi Sun, Seungwook Han, Bowen Tan, Gurpreet Gosal, Xudong Han, Varad Pimpalkhute, Shibo Hao, Ming Shan Hee, Joel Hestness, Haolong Jia, Liqun Ma, Aaryamonvikram Singh, Daria Soboleva, Natalia Vassilieva, Renxi Wang, Yingquan Wu, Yuekai Sun, Taylor Killian, Alexander Moreno, John Maggs, Hector Ren, Guowei He, Hongyi Wang, Xuezhe Ma, Yuqi Wang, Mikhail Yurochkin, Eric P. Xing
arXiv Machine Learning
Sep 21

The Refutation Gap: Certifying Both Halves of an Optimality Claim

The paper addresses the asymmetry in verifying optimality claims for synthesis pipelines, distinguishing between the upper bound (existence of a program) and the lower bound (non-existence of a smaller program). It introduces a pipeline that synthesizes minimal linear straight‑line programs over GF(2) and produces DRAT proofs for every UNSAT result, thereby closing the so‑called refutation gap for 121 previously uncertified optimality claims. The authors report that the median proof size is 1.1 MB, checking takes 1.9× the solving time, and that their verification process uncovered defects missed by code review, highlighted interface obstacles, and exposed a budget‑related audit failure.

By Rohan Pandey
arXiv Machine Learning
Sep 21

Time series generation with spectrally aligned latent flow matching

The paper introduces a spectrally-aligned latent-flow model for time‑series generation that trains the latent space to preserve dynamical properties relevant to synthetic data quality. By incorporating fine‑tuning losses based on Fourier, wavelet, and signature transforms, the method mitigates spectral mismatches caused by latent compression and ensures alignment with true signals in terms of smoothness and targeted spectral content. Experiments on real‑world long‑range univariate and multivariate benchmarks show that the aligned model outperforms a base latent‑flow model and state‑of‑the‑art approaches in signal realism, computational efficiency, and local structure alignment.

By Camilo Carvajal Reyes, Felipe Tobar
arXiv Machine Learning
Sep 21

FairLMs: A Turnkey Library for Fairness in Language Models

FairLMs is a Python library designed to streamline fairness research in language models by unifying bias measurement, mitigation, and evaluation evidence. It offers 33 intrinsic and extrinsic metrics, 14 mitigation components across four intervention categories, 14 diagnostic tools, adapters for major Transformer architectures and hosted APIs, and benchmark loaders. The library enforces explicit declarations of model capabilities and input requirements, ensuring compatibility and reproducibility across components and datasets.

By Jiale Zhang, Michael Larionov, Zichong Wang, Zhipeng Yin, Wenbin Zhang
arXiv Machine Learning
Sep 21

Exemplar Partitioning for Mechanistic Interpretability

The paper introduces Exemplar Partitioning (EP), an unsupervised technique that constructs interpretable feature dictionaries from large language model activations by clustering streamed activations into Voronoi regions defined by exemplars and their averages. EP allows comparison of dictionaries across layers, checkpoints, and architectures, and demonstrates utility in interpreting model behavior, tracking training dynamics, detecting hidden concepts, and enabling targeted interventions. Experiments on Gemma‑2‑2B and Llama‑3.1‑8B show EP can reveal how instruction tuning reorganizes harmful prompt activations, facilitate interventions that alter model responses, and achieve high concept‑detection performance while requiring far fewer construction tokens than comparable methods.

By Jessica Rumbelow
arXiv Computation and Language
Sep 21

CASCADE Against Jailbreaks: Combination Across Stages with Controlled Attack-Defense Evaluation

The paper presents a systematic study of combining defenses against jailbreak attacks on Large Language Models across different pipeline stages. It introduces a standardized evaluation framework that defines attack-success-rate, controls query budgets, and applies explicit fairness rules. Across 19 attacks and 15 defenses, the study finds that no single defense dominates, but carefully selected combinations can provide strong safety with minimal loss of utility, offering practical guidance for layered defense pipelines.

By Jiale Luo, Eric Han
arXiv Machine Learning
Sep 21

OverThink: Slowdown Attacks on Reasoning LLMs

The paper introduces OverThink, a slowdown attack that forces reasoning language models (RLMs) to produce many more reasoning tokens while still giving correct answers. By injecting decoy reasoning problems—such as Markov decision processes, language translation, or graphic comprehension—into the model’s context, attackers can dramatically increase token generation (up to 46× on SQuAD and 17× on coding agents). The study evaluates the attack on both proprietary and open-source RLMs across multiple datasets, explores multimodal and coding‑agent variants, and tests several defenses, concluding that defending against OverThink is challenging and that newer RLMs are even more vulnerable due to higher per‑token costs and increased reasoning token usage.

By Abhinav Kumar, Jaechul Roh, Ali Naseh, Marzena Karpinska, Mohit Iyyer, Amir Houmansadr, Eugene Bagdasarian
arXiv Machine Learning
Sep 21

Recursive Language Models Generalize Out of Domain

The paper investigates how restricting a language model’s view to isolated subtask contexts—via recursive language models—affects learning compared to standard chain‑of‑thought (CoT) approaches. While CoT can simulate recursive reasoning and offers similar in‑distribution performance, it tends to exploit contextual shortcuts that fail out of distribution; recursive models avoid this by isolating context, thereby preventing such shortcuts. The study shows that merely covering the correct rule is insufficient for robust reasoning beyond the training distribution.

By Chenxiao Yang, Zhiyuan Li, David McAllester, Nathan Srebro
arXiv Computation and Language
Sep 21

Cultural Alignment in Large Language Models Using Soft Prompt Tuning

The paper proposes a method for culturally aligning large language models (LLMs) using soft prompt tuning optimized via Differential Evolution (DE). Unlike traditional fine‑tuning or reinforcement learning, this approach keeps model weights frozen and requires no preference data, instead leveraging aggregated survey scores from Hofstede's Value Survey Module (VSM13). Experiments on four countries and four instruction‑tuned models show that DE‑optimized prompts reduce cultural discrepancy, improve agreement with the World Values Survey, and are preferred in blinded pairwise evaluations by LLM judges.

By Reem I. Masoud, Martin Ferianc, Philip Treleaven, Miguel Rodrigues
arXiv Computation and Language
Sep 21

Conformal Privacy Auditing: Calibrated Re-identification Attacks with Statistical Guarantees

Conformal Privacy Auditing (CPA) is a distribution‑free framework that calibrates re‑identification risk for each released document against large language model (LLM)‑empowered adversaries. It outputs a conformal ambiguity set of candidate identities that is guaranteed to contain the true identity with a user‑chosen confidence level under exchangeability, along with an interpretable leakage proxy derived from the set size. CPA supports both logit‑access and sampling‑only attackers, enabling audits of both open‑source and proprietary models, and demonstrates calibrated coverage across various benchmarks and attacker configurations.

By Shuo Huang, Gholamreza Haffari, Xingliang Yuan, Ting Yu, Lizhen Qu
arXiv Computation and Language
Sep 21

Generative Artificial Intelligence Chatbots for Motivational Interviewing: A Scoping Review From System Design to Intervention Outcomes

This scoping review examined 48 studies on generative AI chatbots designed to deliver motivational interviewing (MI). It found that most systems were text‑based and disembodied, with about half incorporating dynamic adaptation, and that safety reporting was inconsistent. While user perceptions were generally positive and many studies reported MI‑consistent interactions, evidence for sustained behavioral or functional change remains limited.

By Runze Hu, Jingqi Kong, Yang Yang, Yihang Yang, Jingyao Liu, Haizhou Tang, Shanghang Zhang, Zheng Liu
arXiv Computer Vision
Sep 21

MultiHU-TD: Multifeature Hyperspectral Unmixing Based on Tensor Decomposition

The paper introduces MultiHU‑TD, an interpretable framework for multifeature hyperspectral unmixing that employs tensor decomposition and incorporates the abundance sum‑to‑one constraint via an ADMM algorithm. It extends previous models by adding mathematical morphology and neighborhood patch analysis, and provides detailed mathematical, physical, and graphical interpretations linked to the extended linear mixing model. Experiments on real hyperspectral images demonstrate the model’s interpretability and effectiveness, with code released on GitHub.

By Mohamad Jouni, Mauro Dalla Mura, Lucas Drumetz, Pierre Comon
arXiv Computation and Language
Sep 21

Transsion's Speaker-Attributed Multilingual ASR System for the MLC-SLM 2026 Challenge

The paper describes Transsion Speech Team’s submission to Task 1 of the MLC‑SLM 2026 Challenge, aiming at speaker‑attributed transcription for multilingual conversational speech. Their cascaded framework includes a DiariZen‑based speaker diarization module, a Qwen3‑Omni‑based long‑form multilingual ASR module with CTC alignment for precise timestamps, and a fusion module that merges diarization and transcription outputs into speaker‑attributed STM results. On the official evaluation set, the system achieved a tcpMER of 15.41% and secured second place among all participants.

By Zhecheng Ren, Xuanji He, Xiaoxiao Li, Zhichen Han, Gaoyang Dong, Gaosheng Zhang, Minchuan Chen, Fengjie Zhu
arXiv Machine Learning
Sep 21

Robust Mixture Models for Algorithmic Fairness Under Latent Heterogeneity

The paper introduces ROME, a framework that learns latent group structure while optimizing worst-group predictive performance. ROME links latent-variable modeling with distributionally robust optimization through an Expectation-Maximization approach for linear models and a neural Mixture-of-Experts for nonlinear settings. Experiments on simulations and three real-world regression datasets show that ROME improves worst-group performance while maintaining competitive overall accuracy compared to existing group-aware and group-label-free robust learning methods.

By Siqi Li, Molei Liu, Yiwei Lyu, Ziye Tian, Chuan Hong, Nan Liu
arXiv Computation and Language
Sep 21

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

The paper investigates gender bias in machine translation evaluation metrics using an occupation-balanced subset of GAMBIT+ across seven English‑source language pairs, including a new German extension. It finds that masculine translations tend to receive higher scores and that biases align with stereotypical gender representations, though the strength varies by evaluator and language. The study highlights that assessing bias requires multiple dimensions beyond a single aggregate measure.

By Orfeas Menis Mastromichalakis, Giorgos Filandrianos, Wafaa Mohammed, Giuseppe Attanasio, Chrysoula Zerva
arXiv Machine Learning
Sep 21

RegKT: Interpretable and Robust Deep Knowledge Tracing With IRT-Regularizer

RegKT introduces an IRT‑regularizer to deep knowledge tracing models, aiming to improve both interpretability and robustness. The paper argues that while deep learning has boosted accuracy, it has reduced explainability and increased overfitting, especially on small educational datasets. By adding this regularization, RegKT seeks to make deep knowledge tracing more suitable for real‑world educational applications.

By Samuel Girard, Juan D. Pinto, Jill-J\^enn Vie, Amel Bouzeghoub
arXiv Machine Learning
Sep 21

Data Attribution via Sketched Metadifferentiation

The paper introduces two algorithms, MAGE and SPELL, that enable efficient data attribution in neural networks by estimating a large influence matrix from a limited number of measurements. These methods leverage existing metagradient techniques without additional computational overhead, addressing the challenge of predicting the impact of removing training data in non‑convex models. Experiments show that MAGE and SPELL outperform current baselines across various training scales and measurement budgets.

By Yuxi Chen, Hamza Golubovic, Han Tong, Arian Maleki, Andrew Ilyas
arXiv Computer Vision
Sep 21

Traits Run Deeper: Trait-Specific Asymmetric Fusion for Multimodal Personality Assessment

Traits Run Deeper introduces a personality assessment framework that tailors multimodal fusion to each trait dimension. It comprises a Multimodal Foundation Representation module that uses psychology-informed semantic templates, a Trait-Specific Modality Fusion module that asymmetrically fuses modalities to reduce cross‑modal interference, and a Distribution‑Calibrated Personality Regression module that corrects label imbalance. The approach achieves a ~25% reduction in mean squared error on the AVI Challenge 2026 validation set and wins the Personality Assessment Track.

By Jia Li, Qian Chen, Wei Wang, Xinyu Li, Zhenzhen Hu, Dongsheng Shao, Richang Hong, Meng Wang
arXiv Machine Learning
Sep 21

Available Guardrails: Certifying Selective Prediction across ML Systems

The paper introduces a method to certify selective prediction in machine learning systems by computing the availability of safety gates through exact-binomial inversion and dynamic programming. It demonstrates that a truth-informed planner can significantly improve mean coverage over naive approaches, and that reallocating error budgets further enhances coverage across diverse applications such as LLM tool‑calling, content moderation, lesion classification, and recommendation. The study highlights the importance of planning and finite‑sample estimation in ensuring reliable, granular deployment of selective predictors.

By Parivesh Priye, Yufeng Wang, Haibin Ling, Michael Chaykowsky