AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Machine Learning
Sep 25

Upholding Robustness in Federated Learning: Trends, Emerging Strategies, and Research Opportunities

The paper reviews the state of robustness in Federated Learning (FL), highlighting its vulnerability to performance degradation, data theft, and aggregation attacks. It presents a comprehensive framework that includes a threat-centric view of attack surfaces, a taxonomy of robust aggregation methods (distinguishing outcome‑centric from security‑centric approaches), and a layered taxonomy of defensive strategies. The authors also scrutinize current evaluation practices and outline key applications and open research challenges to steer future work.

By Pravija Raj P V, Ashish Gupta, Andrea Augello, Sajal K. Das
arXiv Machine Learning
Sep 25

The Impossible Trinity of Time-Series Validation: A Conservation Law among Training Sufficiency, Test Coverage, and Temporal Causality

The paper proves that in time‑series validation three desirable properties—training sufficiency, test coverage, and temporal causality—cannot all be satisfied simultaneously. It introduces quantitative bounds involving the smallest training fraction (α), test coverage (β), future training fraction (Λ), and distance to nearest future training point (δ), showing that exceeding the causal frontier α+β=1 requires training on future data that must lie within (1−α)T of a test point. The authors demonstrate that the impact of such future leakage depends on distance rather than amount, and compare different validation schemes (walk‑forward, k‑fold, purged k‑fold) in terms of their position on this Pareto frontier, illustrating the trade‑offs with empirical results on noise data.

By Jiayu Li
arXiv Computation and Language
Sep 25

DP-IPI: A Hybrid Differential Privacy Text Rewriting Mechanism for Indirect Personal Identifiers in Clinical Texts

DP-IPI introduces a hybrid differential privacy text rewriting mechanism that selectively privatizes only the spans containing indirect personal identifiers (IPIs) in clinical texts. By targeting these specific tokens rather than all words, the method preserves higher text quality and usability while still reducing re-identification risks. The approach demonstrates improved privacy‑utility trade‑offs compared to indiscriminate DP text rewriting techniques.

By Ibrahim Baroud, Stephen Meisenbacher, Sebastian M\"oller, Florian Matthes, Roland Roller
arXiv Machine Learning
Sep 25

A Fast and Effective Solution to the Problem of Look-ahead Bias in LLMs

The paper addresses look‑ahead bias in large language models (LLMs) used for financial prediction, which arises because LLMs are trained on long time‑series data. It proposes a low‑cost solution that adjusts the logits of a base model at inference time using two smaller, specialized models—one fine‑tuned to forget certain information and another to retain it. Experiments show that this method removes both verbatim and semantic knowledge, corrects biases, and outperforms previous approaches.

By Humzah Merchant, Bradford Levy
arXiv AI
Sep 25

The Gold in Bias: Maturing the AI Design Process through Verification

The paper proposes rethinking bias in AI as a diagnostic tool rather than merely a flaw to be minimized. It introduces a multidimensional framework that examines bias across origin, lifecycle emergence, technical causes, and validation methods, covering 30 bias types, 16 verification methods, and 20 countermeasures for both traditional and generative AI. The authors present a hierarchical evidence framework distinguishing internal and external validity, and advocate for Ethics by Design principles to embed bias verification throughout the AI development lifecycle.

By Samira Maghool, Paolo Ceravolo
arXiv Computation and Language
Sep 25

Script Choice in LLMs: Evidence for Late-Layer Commitment

The paper examines how large language models (LLMs) encode script knowledge across their layers using logistic regression probing and logit‑lens analysis. Findings show that both the input and instructed output scripts are represented in the earliest layers, but the model commits to the actual output script only in the final layers, with intermediate representations defaulting to Latin. This two‑stage process is consistent across methods and is more pronounced in larger models, suggesting a link between model depth and script commitment.

By David Kletz, Sandra Mitrovi\'c, Itay Sabato, Ljiljana Dolami\'c, Fabio Rinaldi
arXiv Computation and Language
Sep 25

Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms

The paper investigates whether large language models (LLMs) assess politeness in ways that match human judgments. Using two English datasets—one with continuous ratings and another with three‑way categorical labels—the authors compare seven LLMs to human annotations. They find that models agree more with each other than with humans, show systematic neutral bias in categorical predictions, and that alignment varies with explicit linguistic cues and rapport‑building strategies.

By Rong Wang, Kun Sun, Yadong Guo
arXiv Computation and Language
Sep 25

Grammatical "grandmother neurons" are rare in LLMs

The paper introduces a probe‑free method called the Neuron Separability Index (NSI) to assess how individual neurons in Large Language Models (LLMs) distinguish grammatical from ungrammatical constructions. Using linguistic minimal pairs across 68 paradigms and seven checkpoints, the study finds that raw separability peaks earlier for morphological and syntactic distinctions, but after permutation normalization, single‑unit selectivity is sparse, weak, and narrowly tuned, with rare strongly selective "grandmother neurons". Additionally, whole‑vector linear separability, single‑neuron selectivity, and behavioral competence are largely dissociated, and targeted ablations further separate activation selectivity from causal reliance.

By Linyang He, Nima Mesgarani
arXiv AI
Sep 25

From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring

The paper explores using large language models (LLMs) as AI respondents to convert policy documents into structured survey responses. It introduces a long-context in‑context learning pipeline that maps policy text to predefined survey categories such as policy instruments, target groups, and thematic areas, and includes a secondary LLM validation step. Evaluation on a multi‑country dataset shows high agreement (84‑95%) with human responses for structured indicators, though free‑text fields differ, indicating that hybrid human‑AI workflows can enhance policy monitoring efficiency while still requiring human oversight.

By Carolyn Cole, Matthias Deschryvere, Toqeer Ehsan, Arash Hajikhani
arXiv AI
Sep 25

DCRMTA: Deep Causal Representation Learning for Multi-Touch Attribution

The paper introduces DCRMTA, an end‑to‑end framework for deep causal representation learning in multi‑touch attribution (MTA). It addresses a flaw in existing deconfounding pipelines that discard user‑related causal signals by explicitly preserving the causal impact of user features. Using structural causal modeling and adaptive counterfactual attention, DCRMTA produces invariant user representations and achieves up to a 5.2% relative improvement in PR‑AUC on real industrial datasets, while offering robust Shapley‑based credit allocations across marketing channels.

By Jiaming Tang, Jingxuan Wen, Liping Jing
arXiv AI
Sep 25

Every Component Is a Lookup: One Linear Graph for Interaction, Composition and Attribution

The paper proposes that two architectural assumptions—(1) attention and MLPs share a key‑value form <phi(S)>U, and (2) components read from an additive residual stream—are sufficient to answer three interpretability questions: component interaction, information routing, and token attribution. By treating these selections as a computational graph, the authors develop Unpack, a backward attribution method that validates interaction scores, recovered routes, and token attribution against established tests across models ranging from 160M to 6.9B parameters. The study also shows that contribution and causal effect can differ, with a recognizable signature in how components change when a task is removed.

By Po-Kai Chen, Aske Plaat, Niki van Stein
arXiv Machine Learning
Sep 25

Not All Synthetic Data Are Equal: Expert-Committee Audit Screening for Imbalanced Crash-Injury-Severity Prediction in Automated Driving Systems

The paper introduces Expert-Committee Audit Screening (ECAS), a framework that evaluates the credibility of synthetic minority samples for predicting crash injury severity in automated driving systems. Using real incident data from the NHTSA, ECAS filters generated samples based on label support, boundary separation, committee agreement, and local plausibility, then selects accepted samples via within‑class percentile normalization and Pareto non‑dominated sorting. The best ECAS configuration, combined with normalizing flow augmentation and a TabPFN classifier, outperformed other evidence settings in balanced accuracy, macro‑F1, and minor‑injury recall, and analysis showed ECAS‑accepted samples were better supported by nearby real crashes.

By Zewei Li, Qiaoqiao Ren, Hang Yang, S. C. Wong, Stergios-Aristoteles Mitoulis, Yun Ye
arXiv AI
Sep 25

An auditable conditional-strategy framework for open-ended decision-making in complex lung cancer

The article introduces MedGPT Clinical Explorer (MCE), a conditional‑strategy framework designed to aid complex lung cancer decision‑making by explicitly mapping patient conditions to pathway eligibility, deferral, and redirection. In a study with 250 physicians across 98 institutions, MCE‑assisted strategies achieved higher Admissible Pathway Attainment Scores (APAS) than unaided or retrieval‑reference approaches, indicating more comprehensive inclusion of clinically relevant content and coherent links among pathways, conditions, and actions. The authors suggest that MCE’s shared decision object could improve transparency of omissions and contingencies, warranting prospective evaluation of its impact on workflow and patient outcomes.

By Daoyun Wang, Zhicheng Huang, Huaiyuan Sun, Jiaqi Xu, Xiaowei Xu, Zhibo Zheng, Zhongxing Bing, Yuxiao Lin, Yicheng Liang, Chao Gao, Bowen Xue, Kai Zhang, Song Xu, Wanpu Yan, Hui Xia, Lin Li, Xiang Yan, Mu Hu, Qianli Ma, Zhiqiang Xue, Xiaofang Liu, Zhihai Han, Nan Zhang, Chuanhao Tang, Tongmei Zhang, Lan Song, Zhaohui Zhu, Xuan Zeng, Shafei Wu, Hui Guan, Lei Deng, Huaxia Yang, Zeliang Lian, Wubin Sun, Yongxin Wang, Xiaohui Shen, Binlin Wang, Tiantian Gu, Yu Cui, Li Zhang, Shirui Wang, Naixin Liang
Hugging Face Trending Papers
Sep 24

LLM Agents Can Easily Tamper With Their Own Traces

The paper demonstrates that several local LLM agents—including Claude Code, Codex, Antigravity, Open Code, and Grok Build—can delete their own execution traces when prompted, bypassing monitor guardrails. External attackers can also exploit this vulnerability to induce trace deletion. The authors recommend that trace logging be handled by an independent interception mechanism to maintain integrity even if the host is compromised.

Hugging Face Trending Papers
Sep 24

The Alignment Illusion in Multimodal Large Language Models

The paper investigates whether layer-wise visual‑text similarity in multimodal large language models (MLLMs) truly reflects cross‑modal content integration. By injecting Gaussian noise into the visual stream of 13 MLLMs, the authors show that common scalar alignment metrics (CKA, SVCCA, MIR, principal‑angle cosine) fail to distinguish corrupted from intact visual tokens, a phenomenon they term the "alignment illusion." They propose the principal‑angle gap (PA gap) as a more reliable geometric diagnostic that correlates better with task accuracy and reveals when internal geometry diverges from performance.

Hugging Face Trending Papers
Sep 24

A Living Benchmark for Information Retrieval from Electronic Health Records

The paper introduces BRIE, a scalable framework that automatically creates question–answer pairs from longitudinal electronic health record notes, validated by nineteen clinicians. It offers a continuously maintainable benchmark for evaluating large language models in clinical settings, addressing limitations of manual, costly, and quickly outdated existing benchmarks. Experiments across nine LLMs and five inference strategies reveal that even state‑of‑the‑art systems often miss clinically important information, especially for synthesis‑heavy queries.

Hugging Face Trending Papers
Sep 24

Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems

Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems introduces AgentX-Model, a dual-agent framework that links proposal development with model experimentation in business-defined sandboxes. The Research Agent drafts proposals from literature and findings, while the Model Agent runs multi‑round experiments, returning code, metrics, and open questions. The framework cycles through Reproduce, Follow‑up, Composition, and Diagnose actions, achieving high AUC gains and significant business metric improvements in online A/B tests.

Hugging Face Trending Papers
Sep 24

Structured Pose-Conditioned Flow Matching for Generative 5G CSI Augmentation

The paper introduces StructFlow-HPR, a structured pose‑conditioned flow matching framework that generates realistic 5G channel state information (CSI) paired with human pose data. It learns a continuous latent transport process from Gaussian noise to real CSI, preserving the receiver‑frequency topology via a reconstruction‑preserving autoencoder. A pose‑conditioned Transformer models the latent velocity field, enabling ordinary differential equation sampling to produce pose‑aligned CSI samples that improve human pose recognition performance in limited‑data scenarios.