AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv AI
Sep 24

Finite-Sample Probabilistic Safety Certification for AI-Based Grid-Edge Coordination

The paper introduces a finite‑sample probabilistic safety certification framework for black‑box AI decision models used in closed‑loop grid operation. It transforms the AI‑grid evaluation into a binary unsafe outcome under a safety specification and applies exact binomial inference to provide a tight one‑sided upper bound on the unsafe operation probability, using held‑out calibration scenarios. The framework also incorporates physically interpretable sample‑space adversarial attacks to address distribution shifts and is validated through case studies involving 1,000‑agent AI models for grid‑edge flexibility coordination.

By Yihong Zhou, Hanbin Yang, Thomas Morstyn
arXiv AI
Sep 24

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

The paper introduces the Systemic Risk Index, an open pipeline and dashboard that aggregates evidence from 19 public AI benchmarks into four systemic‑risk categories defined by the EU GPAI Code of Practice. It evaluates 18 models using harm‑preserving perturbations and simulated deployment contexts, offering users the ability to switch between average and worst‑case aggregation and to trace each risk rating back to its benchmark evidence. The study finds that worst‑case scores can be 14 to 37 points lower than average scores, and that LLM judges agree with human graders at a level comparable to human‑human agreement.

By Jacob T. Emmerson, Phuong-Anh Nguyen-Le, Ronan Romano, Wilber Sean V. Anterola, Yann Billeter, Zhijing Jin
arXiv AI
Sep 24

What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus

The paper investigates why some tasks in the Terminal‑Bench/Frontier‑Bench datasets fail for all agents, distinguishing genuine difficulty from artifacts such as missing context, broken solutions, infrastructure failures, or verifier bypasses. Analyzing 125 all‑fail tasks, only 78 are certified as genuinely unsolved after applying a validity screen; the rest are attributable to broken oracles, infrastructure issues, bypassable verifiers, or insufficient evidence. The study concludes that a zero pass rate does not automatically indicate a hard task and recommends that frontier benchmarks provide evidence for all‑fail tasks before claiming capability gaps.

By Edward Lue Chee Lip, Boden Moraski, Tim Knappe, Lang Xiong, Sarvesh Gharat, Antonio Mari, Ivan Bercovich
arXiv AI
Sep 24

COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluation

COPE (Continual Optimization with Personalized embedding and self-Evaluation) is a new framework that continually personalizes large language models using learnable user embeddings and self‑evaluation to generate proxy rewards. It integrates preference capture, self‑evaluation calibration, and personalized response optimization into a single update step, allowing continuous model updates even when explicit user feedback is sparse. Experiments demonstrate that COPE outperforms both training‑free and training‑based baselines, remains complementary to Retrieval‑Augmented Prompting, and shows reliable self‑evaluation, meaningful preference patterns, stable general capabilities, and robustness to shifting preferences and alternative evaluators.

By Ruike Cao, Fugen Yao, Liang Dong, Jian Xu, Guanjun Jiang, Li Xiao
arXiv AI
Sep 24

Cross-Modal Contrastive Learning from Histopathology and CT for Automated Renal Cell Carcinoma Grading

The study introduces RCC-Align, a cross‑modal contrastive learning framework that aligns paired histopathology whole‑slide images and CT scans to enhance noninvasive grading of clear cell renal cell carcinoma (ccRCC). Using patient‑level five‑fold cross‑validation on TCGA and CPTAC cohorts, RCC‑Align achieved an AUC of 0.601 and AUPRC of 0.599 for low‑ versus high‑grade ccRCC classification, outperforming CT‑only baselines and showing stronger WSI‑CT embedding alignment. The approach relies solely on CT at inference, potentially aiding grading when biopsy is unsafe or limited by tumor heterogeneity.

By Amit Das, Tanmay Shukla, Naofumi Tomita, Faraz Farhadi, Jessica Sin, Ari Hakimi, Chad Vanderbilt, Jie-Fu Chen, Ritesh Kotecha, Weijie Ma, Bing Ren, Saeed Hassanpour
arXiv AI
Sep 24

When Clients Are Orchestrated: Strategic Gradient Manipulation to Defeat Federated Learning Servers with Efficient Defense

The paper introduces Fed-ADR, a coordinated attack framework where a malicious orchestrator server directs heterogeneous adversarial clients to adapt their gradient updates in real time, thereby evading existing federated learning defenses and drastically reducing global model accuracy. It also presents a lightweight detection mechanism that estimates true client gradients from historical data to spot coordinated attacks, and an in-situ recovery method that restores model performance without restarting training. Experiments on MNIST, Fashion‑MNIST, and CIFAR‑10 show the attack can drop accuracy from over 90% to below 10%, while the defense can recover accuracy to above 90% within a few rounds at a computational cost at least 20× lower than retraining from scratch.

By Mohamed Shaaban, Ahmed Abdelnaby, Mohamed Elmahallawy
arXiv AI
Sep 24

The Linear Representation Hypothesis Needs a Group Action

The paper argues that the Linear Representation Hypothesis (LRH) should not be treated as a single claim but as a family of claims differentiated by how representations are considered equivalent. It highlights that different equivalence notions preserve different structures, leading to metrics, probes, and interventions that may actually test distinct hypotheses. By formalizing these ideas with group actions, the authors provide a framework that clarifies how assumptions vary across metrics, reading points, and analysis stages, and they apply it to audit common representation quantities and recent interpretability analyses.

By Louie Hong Yao, Yuhao Li, Shengchao Liu
arXiv AI
Sep 24

Beyond the Illusion of Power: Calibrating Quasi-Experiments in Observational IS

Information systems researchers increasingly rely on quasi‑experimental methods such as difference‑in‑differences and instrumental variables to infer causal effects from observational panel data. A large Monte Carlo study of 9,837 parameter settings (≈9.8 million simulated datasets) shows that the gap between planned and achieved power is largely driven by serial correlation, panel attrition, staggered adoption bias, and parallel‑trend pre‑testing—factors that no closed‑form power calculator can fully capture. For IV designs, increasing sample size does not improve power or reduce exclusion bias unless instrument strength is enhanced, underscoring that identification hinges on the instrument rather than on larger N.

By Spandan Ghose Chowdhury
arXiv AI
Sep 24

Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning

The paper introduces Safety to Competence (S2C), a two‑stage reinforcement learning framework that first learns a safety filter and then trains a competitive task policy while embedding the filter. By separating safety synthesis from task learning, S2C reduces training complexity and prevents the policy from being exploited by adversarial attacks. Experiments on simulated touchdown games show that S2C achieves higher win rates, better Elo ratings, and lower exploitability than eight safe‑RL baselines, and hardware tests confirm its competence against a human opponent.

By Ruihan Wu, Rui Yang, Donggeon David Oh, Duy Nguyen, Haimin Hu
arXiv AI
Sep 24

InfiNoVA: Infinite Novel View Augmentation for Viewpoint Invariant Robot Policies

InfiNoVA is a data‑augmentation framework that transforms synchronized multi‑camera demonstrations into a dense, geometrically consistent set of training views by reconstructing each manipulation trajectory as a time‑varying 3D Gaussian. The method renders novel observations from sampled camera poses while preserving the original state‑action pairs, improving frame‑level fidelity and temporal consistency compared to generative synthesis. Across four real‑world manipulation tasks, policies trained with InfiNoVA achieve 5.4× higher average success under unseen randomized viewpoints than VISTA‑based augmentation and 1.7× higher success than training on all five physical camera views.

By Sai Puneeth Reddy Gottam, Elmar Rueckert, Vedant Dave
arXiv AI
Sep 24

LabourCrew: A Multi-Agent RAG Framework for Trustworthy Adversarial Deliberation and Statutory Reasoning over Labour Law

LabourCrew is a multi‑agent Retrieval‑Augmented Generation (RAG) framework designed for trustworthy statutory question answering in labour law. It introduces three grounding mechanisms: StatuteGraph, an evidence‑exchange ledger, and a calibrated trust gate that controls false‑accept rates. Evaluated on a Bangla Labour Act QA set, LabourCrew achieves a false‑accept rate of 0.081 and higher answer relevancy than existing RAG methods, demonstrating that calibrated abstention is key to auditable legal QA.

By Fatema Tuj Johora Faria, Mukaffi Bin Moin, Jubayer Al Mahmud, M. F. Mridha, Md. Alam Hossain
arXiv AI
Sep 24

RelCheck: Dual-Evidence Spatial Grounding for VLM Hallucination Correction

RelCheck is a training‑free post‑hoc correction pipeline that addresses relational hallucinations in multimodal large language models. It augments object‑level visual grounding with two forms of relational evidence—learned scene‑graph triples from RelTR and deterministic spatial predicates derived from bounding‑box geometry—forming a three‑layer visual knowledge base. When applied to LLaVA v1 13B, RelCheck improves the overall MME hallucination score from 585.0 to 630.0, with the most significant gain on spatial position accuracy.

By Siddhi Patil, Navrati Saxena, William B. Andreopoulos
arXiv AI
Sep 24

Controlled Attribute-Specific Summarization of Interrogative Dialogues

The paper introduces CASPER, a Chain-of-Thought Attribute-Specific Prompting framework designed to generate accurate, coherent, and attribute-relevant summaries of interrogative dialogues. It leverages structured prompting, iterative refinement, and a hierarchical evaluation mechanism called RoleEval to improve factual consistency and contextual completeness. The authors also present MINDSum, a new dataset of 6,000 annotated utterance pairs, and show that CASPER outperforms existing summarization models on ROUGE, BERTScore, and human expert evaluations.

By A Aditya Bhardwaj, Arjit Singh Arora, Md Shad Akhtar
arXiv AI
Sep 24

Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing

The paper examines whether state‑of‑the‑art large language models (LLMs) produce feedback that aligns with expert teachers’ pedagogical practices, focusing on feedback type and adaptivity. Using a refined taxonomy of seven feedback focus types, the authors annotate and compare teacher and LLM‑generated feedback from three university writing courses, creating the FeedType benchmark. Their analysis shows that while most LLMs cover many feedback types, they do not match teachers’ distribution patterns or adaptive behavior across draft stages and student performance levels.

By Norah Almousa, Shayan Peyghambari Oskoui, Raquel Coelho, Gayle Rogers, Xiang Lorraine Li, Diane Litman
arXiv AI
Sep 24

"We'll Fix It Later": Education, AI, and the Deferral of Privacy in EdTech

The article "We'll Fix It Later": Education, AI, and the Deferral of Privacy in EdTech examines how educational technology platforms collect sensitive student data but often postpone privacy considerations until later stages of product development. Through 12 interviews and a policy audit of 48 platforms, the study finds that privacy is acknowledged yet deferred in favor of functionality, growth, and funding, with responsibility frequently shifted to cloud providers or downstream institutions. The audit reveals that many platforms lack clear AI disclosures and provide limited governance details, indicating a gap between data collection practices and privacy accountability.

By Meghna Manoj Nair, Rachel Greenstadt
arXiv AI
Sep 24

From Agent Output to Authorized Transition

The paper introduces the Agile‑V Assurance Spine, a cross‑domain transition contract designed to manage the assurance of outputs from agentic engineering systems across software, firmware, and PCB domains. It specifies that evidence is only accepted when it demonstrates required properties through an authoritative source profile, is tightly bound to the exact artifact and policy baseline, stays current with declared dependencies, and meets risk‑appropriate independence and authority. Gate decisions are recorded as receipts, approvals and exceptions are scope‑ and time‑bounded, and authorization is rechecked at the effect boundary before any merge, deployment, flashing, release, or fabrication step.

By Christopher Koch
arXiv Machine Learning
Sep 24

RL Starts before RL: On Policy Distillation for Better Reinforcement Learning

The paper investigates on‑policy distillation (OPD) as a preparatory step for reinforcement learning (RL). It shows that students initialized with OPD achieve higher final RL performance than those trained directly with RL or with supervised fine‑tuning followed by RL, even when OPD offers little immediate accuracy gain. The study also finds that the choice of distillation objective (reverse‑KL vs forward‑KL) and the source of trajectories influence OPD’s effectiveness at different stages of RL training.

By Shuai Dong, Yongfu Zhu, Yuqi Xu, Weichu Xie, Liuwenpu, Ziyue Wang, Kaiwen Tuo, Congcong Wang, Siyuan Wang, Wenqi Shao, Shuai Yang, Ji Zhao, Caoyuan Ma, Wenzheng Chang, Taiqiang Wu, Xinlei Yu, Hongrui Wu, Xiaoxuan He, Fangke Chen, Dianyi Wang, Kanghui Tian, Sirry Chen, Xingyu Liu, Xiangnan Wu, Jiawei Guo, Haowen Hou, LingHan Chen, Zhongyu Wei, Jiaqi Wang
arXiv AI
Sep 24

Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis

The study examines how different editorial framings in prompts influence large language models’ statistical analysis reports. Using a 4×4 factorial design, researchers found that certain framings—particularly brutally critical prompts on genuine effects and significance-seeking prompts on underpowered nulls—led to factual misrepresentations. Tone shifts were more widespread, with critical framing inducing defensive language across all data patterns, while a confound in the data largely prevented both factual and tonal distortions.

By Paras Balani, Subhrakanta Panda
arXiv AI
Sep 24

Loss Choice or Model Choice? The Role of Forecast Level in Cryptocurrency Volatility Forecasting

The paper investigates how the choice of loss function versus the choice of forecasting model affects cryptocurrency volatility predictions. By comparing seven loss functions and five models, and aligning forecast levels before evaluation, the study finds that after level adjustment model choice dominates performance differences, while loss-induced variations largely disappear. The work clarifies that apparent loss effects in raw comparisons are largely due to forecast level differences rather than intrinsic model performance.

By Andrzej Tokajuk, Jaros{\l}aw A. Chudziak
arXiv AI
Sep 24

Meet, Compare, or Abstain: LatWeave for Deterministic Multi-Hop Question Answering on Knowledge Lattices

LatWeave is a deterministic multi‑hop question‑answering framework that structures knowledge into a multidimensional lattice and reduces QA to three operators—meet, compare, and abstain—while limiting LLM use to extraction and planning. It achieves near‑lossless performance on complete knowledge benchmarks (e.g., MetaQA) and strong results on templated multi‑hop datasets (e.g., 2WikiMultihopQA), while transparently handling incomplete knowledge through abstention. The approach offers reproducible, auditable answer paths with no performance penalty within its operating envelope.

By Yuze Ren, Shaoheng Fan, Tao Wang, Yabo Yan, Han Han