AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,808 stories · RSS feed

arXiv Machine Learning
Sep 3

Eliciting ESG Preferences for Reinforcement Learning-Based Portfolio Optimization

The paper proposes a Multi-Objective Reinforcement Learning framework for portfolio optimization that incorporates ratings from three ESG agencies, addressing the divergence in ESG rating methodologies. It couples this with a Preference Elicitation system using Gaussian Processes, allowing users to infer latent utility functions via pairwise comparisons of portfolios based on Sharpe ratios and ESG scores. Experiments with LLM-generated portfolio managers show that regional background influences preference weights, with European personas prioritizing ESG alignment and Texas personas favoring risk‑adjusted returns.

By Giovanni Dispoto, Marcello Restelli, Carmine Ventre
arXiv Machine Learning
Sep 3

CodePoisonRAG: Knowledge Poisoning Attacks on Retrieval-Augmented Code Generation

The paper introduces CodePoisonRAG, a framework that poisons retrieval-augmented code generation systems by transforming benign code artifacts into malicious ones. It injects CWE-specific vulnerabilities and false safety claims into a single task-matched artifact, achieving high success rates across multiple generators and even against a defense system. The study demonstrates that attackers can target and propagate specific weaknesses without altering the underlying language model.

By Varun Gadey, Ziad Marey, Alexandra Dmitrienko
arXiv Machine Learning
Sep 3

Full-Model Optimality for Tunable Linear Generative Priors in Compressed Sensing

The paper investigates compressed sensing using a family of tunable linear generative priors linked through their singular value decompositions. It proves that in noiseless Gaussian compressed sensing, the full-dimensional linear prior yields the lowest expected reconstruction error across the entire family, meaning lower-complexity priors do not improve performance in this idealized setting. This contrasts with denoising, where lower complexity priors can reduce error due to bias‑variance tradeoffs, suggesting that the experimental gains seen with neural network priors stem from their nonlinearities.

By Zhaoming Li, Paul Hand
arXiv Machine Learning
Sep 3

Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning

The paper introduces Exchange Policy Optimization (EPO), a framework for semi‑infinite safe reinforcement learning that handles infinitely many constraints by iteratively solving finite subproblems. EPO expands or deletes constraints based on tolerance violations and Lagrange multipliers, maintaining computational tractability while converging to an optimal policy with bounded safety violations. The authors prove finite convergence, provide iteration bounds, and quantify the suboptimality gap under mild assumptions.

By Jiaming Zhang, Yujie Yang, Haoning Wang, Liping Zhang, Shengbo Eben Li
arXiv Machine Learning
Sep 3

Gradient Prediction with Control Variates in the Cheap-Forward Regime

The paper investigates whether idle inference resources can help cut the high cost of scarce GPU usage during training. Using a simulated compute ledger that bills fleet work at a fraction of a GPU forward pass, the authors propose an algorithm that predicts gradients with a low‑precision, inference‑style reverse‑mode program and then refines these predictions with a few exact gradients via a control variate, turning approximation error into variance rather than bias. Experiments on a 124‑million‑parameter language model and across models ranging from 10 M to 774 M parameters show that the method can reduce simulated ledger cost when fleet work is cheap, though it also exhibits both successful transfers and failures, and does not evaluate inference‑only hardware or full optimizer‑by‑batch‑size sweeps.

By Kamil Ciosek, Nicol\`o Felicioni, Juan Elenter, Ehsan Imani
arXiv Machine Learning
Sep 3

Objective-Behavior Alignment: Diagnostics for MORL Policy Selection

The paper introduces a diagnostic workflow for multi‑objective reinforcement learning (MORL) that reveals behavioral differences among policies on the Pareto front, which are not apparent from value vectors alone. It offers quantitative and visual tools to inspect these variations and demonstrates their effectiveness on both simple grid tasks and more complex continuous‑control benchmarks.

By Antonio Mone, Zuzanna Osika, Florian Felten, Pradeep K. Murukannaiah, Mark Fuge, Frans A. Oliehoek, Luciano Cavalcante Siebert
arXiv Machine Learning
Sep 3

Feature Interaction Modeling for Neural Operators

The paper introduces the Feature Interaction Modeling Operator (FM-Operator), a point‑wise query neural operator that explicitly designs feature construction and multiplicative interactions between sensor observations and query coordinates. By reinterpreting the DeepONet aggregation as a diagonally constrained multiplicative interaction, FM-Operator restructures both feature construction and interaction to enable richer information exchange while maintaining point‑wise evaluation. Experiments on several PDE benchmarks show that FM-Operator consistently outperforms vanilla DeepONet and improves upon the Shift‑DeepONet baseline, indicating that tailored representation construction and interaction can enhance query‑based neural operator performance.

By Quan Gu, Xiaoduo Li, Hongxia Liu
arXiv Machine Learning
Sep 3

Adversarial Stress Testing of Outlier Detection in Subjective Image Quality Assessment

The paper introduces a general empirical worst‑case framework for testing outlier‑detection methods in subjective image quality assessment. It presents adversarial attack generators that optimize ratings to maximize the discrepancy between mean opinion scores (MOS) and ground truth, applying these to both discrete and continuous rating scales. The study evaluates several hard and soft outlier‑detection techniques, revealing significant differences in worst‑case performance and proposes low‑complexity methods that achieve strong empirical results.

By Dietmar Saupe
arXiv Machine Learning
Sep 3

Probing Cultural Signals in Large Language Models through Author Profiling

The study investigates cultural biases in large language models (LLMs) by testing their ability to perform author profiling—inferring singers’ gender and ethnicity—from song lyrics in a zero‑shot setting. Evaluating over 10,000 lyrics across several open‑source models, the authors find that most LLMs default toward North American ethnicity, while DeepSeek‑1.5B leans toward Asian ethnicity, and that Ministral‑8B exhibits the strongest ethnicity bias whereas Gemma‑12B is the most balanced. The paper introduces two fairness metrics, Modality Accuracy Divergence (MAD) and Recall Divergence (RD), to quantify these disparities and provides code and results publicly on GitHub and HuggingFace.

By Valentin Lafargue, Ariel Guerra-Adames, Emmanuelle Claeys, Elouan Vuichard, Jean-Michel Loubes
arXiv Machine Learning
Sep 3

Stabilizing Private LASSO under Heterogeneous Covariates via Anisotropic Objective Perturbation

The paper investigates high‑dimensional LASSO under differential privacy using objective perturbation when covariates have heterogeneous scales. It introduces a Gram‑based anisotropic objective perturbation that counteracts the distortion caused by covariate heterogeneity, restoring isotropy in the estimation process. Through an Approximate Message Passing framework and state evolution analysis, the authors show that this approach stabilizes convergence and improves both statistical efficiency and privacy performance compared to standard uniform noise injection.

By Haruka Tanzawa, Ayaka Sakata
arXiv Computation and Language
Sep 3

Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos

The paper presents a semester-long deployment of the VideoPoints platform, featuring a retrieval‑augmented chatbot that answers questions using only the active course’s lecture materials and provides clickable, timestamped citations. Across 833 interactions, 70.5% of messages included citations, no cross‑course references were made, and the bot declined to answer when no lecture evidence matched. The study also shows that the system improves correct‑lecture retrieval by 6.3 percentage points over dense‑only retrieval on the EduVidQA benchmark.

By S M Masrur Ahmed, Jaspal Subhlok
arXiv Computation and Language
Sep 3

GAPS: Dimension-Level Gates for Conditional Activation Steering

The paper introduces GAPS, a dimension‑level gating approach for activation steering in language models. GAPS uses two training‑free gates—a static separability gate based on AUROC and a dynamic posterior gate based on a Gaussian model—to selectively apply steering vectors only to neurons that carry reliable concept information or are currently mis‑activated. Experiments on Gemma‑3 and Qwen‑3 show that GAPS improves or matches the performance of token‑level methods, notably reducing Gemma‑3’s toxicity rate from 6.52% to 0.48% under a fixed capability budget.

By Moghis Fereidouni, Muhammad Umair Haider, Hassan Sajjad, A. B. Siddique
arXiv Computation and Language
Sep 3

PragAlign: Feedback-Guided Pragmatic Alignment for Controlled Synthetic Dialogue Generation

PragAlign is a feedback‑guided framework that improves synthetic dialogue generation by iteratively generating, evaluating, and revising conversations to meet specified service context, target intent, and target emotion. Using an LLM‑based evaluator that scores intent alignment, emotion alignment, coherence, fluency, and overall quality, PragAlign achieves a 99.50% acceptance rate on 800 dialogue specifications, outperforming one‑shot and repeated generation without feedback. Human evaluation confirms that intent expression and dialogue flow are reliably recognized, while emotion appropriateness remains more variable.

By Smitha Muthya Sudheendra, Jaideep Srivastava
arXiv Computation and Language
Sep 3

When Persona Attributes Improve Population Alignment in Large Language Models

The paper investigates how persona prompting—using short textual descriptions of individuals—to align large language models (LLMs) with human survey responses. It examines the impact of selecting different persona attributes and finds that not all attribute combinations improve performance, suggesting that the variation in human responses to survey questions may explain mixed results. The study evaluates multiple attribute selection methods across four social surveys, two countries, six LLMs, and twenty prediction tasks, offering guidance on when persona prompting is beneficial and which attribute choices are most effective.

By Leon Fr\"ohling, Jens Rupprecht, Markus Strohmaier, Claudia Wagner
arXiv Computation and Language
Sep 3

CORAL: An LLM-Native Harness for Production Recommender Systems

CORAL is an LLM‑native harness that automates continual optimization of production recommender systems. It operates in a closed loop: an agent observes system signals, reasons over past decisions, and uses tools—including a numerical optimizer—to reconfigure the recommender while staying within a fixed operating budget. In A/B experiments on two large social platforms, CORAL improved engagement without extra serving cost on one platform and reduced serving cost without harming engagement on the other, demonstrating that a single agentic loop can replace manual engineering for ongoing system tuning.

By Muhammad Rafay Azhar, Yuhang Zhou, Gilbert Jiang, Yuchen Wang, Rahul Sharma, Matthew DeSousa, Jiayi Liu, Xin Guo, Lizhu Zhang, Xiangjun Fan
arXiv Computation and Language
Sep 3

Whose Judgments Count? Representation Gaps in Crowdsourced Content Moderation Produce Unequal Protection from Perceived Toxicity

The study investigates how the demographic makeup of crowdsourced moderators influences who is protected from perceived toxic content online. Using data from 16,221 U.S. respondents who evaluated over 100,000 comments from Twitter, Reddit, and 4chan, the authors find that moderators tend to protect users who share their own demographic identities, a pattern that is amplified when moderator pools mirror the demographics of Prolific participants. Even fully representative moderator groups fail to provide equal protection, leaving Black and LGB users underprotected unless they are overrepresented.

By Zhaodi Chen, Byungkyu Lee
arXiv Computation and Language
Sep 3

Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving

The paper surveys the transition from textual chain-of-thought reasoning to action-grounded reasoning in autonomous driving, highlighting that driving decisions require continuous actions that mirror the spatiotemporal structure of the physical world. It reviews 171 papers, categorizing 130 methods into four main types—language-based, visual-spatial, latent-dynamic, and externalized reasoning—along with 13 subtypes linked to specific regions of interest. The authors argue that the future of driving agent reasoning lies in intermediate representations that are grounded in reality, linked to real-time actions, and verifiable within safety-critical systems.

By Zhengxu Tang, Xiaozhou Zhang, Guofeng Cui, Ziyu Gong, Zi Wang, Yunfei Shi, Ruifeng Deng, Chengzhi Qi, Ke Chen, Sachin Patil, Tianjun Xiao, Langechuan Liu, Pichao Wang
arXiv Computation and Language
Sep 3

Counter-GEO-Bench: Evaluating Defenses Against Information-Distorting Generative Engine Optimization

Counter‑GEO‑Bench is a new benchmark that evaluates how well defenses can stop large language models from producing misinformation when faced with generative engine‑optimized (GEO) content. It contains 247 human‑verified queries paired with both information‑preserving and information‑distorting GEO rewrites, and measures attack success rate, false positives, and answer quality across three victim LLMs. The study shows that existing off‑the‑shelf defenses reduce attack success by at most 5.7 %, while a lightweight baseline called C‑GEO Guard cuts success by 47.6 % with minimal loss of utility.

By Bing Zheng, Zongyao Zhao, Wenming Yang
arXiv Computation and Language
Sep 3

SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation

SignBind-LLM introduces a modular framework for sign language translation that separates continuous signing, fingerspelling, and lipreading into dedicated expert streams. Each expert is pre‑trained independently on about two million pseudo‑gloss sequences, eliminating the need for manual gloss annotation. A lightweight transformer fuses the expert outputs, and a pre‑trained language model converts the fused pseudo‑glosses into fluent English, achieving state‑of‑the‑art performance on multiple benchmarks with lower training cost.

By Marshall Thomas, Edward Fish, Richard Bowden
arXiv Computation and Language
Sep 3

Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews

The paper investigates language-of-study (LoS) bias in NLP peer reviews, defining and distinguishing negative and positive forms of bias. Using a new dataset, LOBSTER, and an LLM-based detection pipeline, the authors analyze 15,645 reviews and find that non‑English papers experience significantly higher bias rates, with negative bias outweighing positive bias. They further identify four subcategories of negative bias, noting that demanding unjustified cross‑lingual generalization is the most common.

By Ehsan Barkhordar, Abdulfattah Safa, Verena Blaschke, Erika Lombart, Marie-Catherine de Marneffe, G\"ozde G\"ul \c{S}ahin