AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv AI
Oct 31

Judging Before Propagation: Task-Graph Contextual Integrity for Multi-Agent Privacy

The paper introduces Task-Graph Contextual Integrity (TG‑CI), a formalism that extends traditional Contextual Integrity to multi‑agent systems by modeling workflows as Multi‑Agent Task Graphs (MATGs). TG‑CI evaluates privacy not just at individual disclosures but across the entire downstream propagation, offering two enforcement policies—Full‑Scope and Auto‑Region—that balance privacy protection with task performance. A new benchmark, MAPLE‑Bench, demonstrates that TG‑CI can cut unauthorized disclosures by up to 94% while preserving over 94% of task accuracy compared to existing local‑history CI methods.

By Hengyu Wu, Blaise Delattre, Yang Cao
arXiv AI
Oct 31

Generative artificial intelligence prompting strategies among Chinese university students across academic tasks

The study surveyed 234 Chinese university students to assess how they use generative artificial intelligence (GenAI) in academic tasks and their confidence in prompting these tools. Results show students frequently employ GenAI across various activities, feel confident especially in providing context and refining prompts, but their strategies vary little across task types, with creative scenarios favoring output‑format specification. Concise instructions and comprehensive initial information are common, while fixed templates are less popular, and perceived cognition most strongly predicts prompting capability.

By Lei Fan, Fangxue Liu
arXiv AI
Oct 31

CovFedAvg: Local Low-Loss Curves and Federated Model Selection

CovFedAvg is a federated learning method that preserves local low‑loss curves and selects a shared model without requiring centralized calibration data or client‑to‑client sample exchange. It aggregates gradient contributions to fit a common endpoint and guarantees that the chosen model’s evaluated training loss is no higher than the previous model or the current‑round FedAvg candidate. Experiments on synthetic, activity‑recognition, and gas‑sensor data show modest accuracy gains over standard FedAvg, with a 0.96% improvement in macro balanced accuracy for a width‑16 network.

By Federico Lucchetti, Marcus V\"olp
arXiv AI
Oct 31

Spectral Invariant Fusion and Refinement for Domain Generalized Semantic Segmentation

The paper introduces SIFR, a frequency-aware dual‑stream framework for domain‑generalized semantic segmentation. SIFR performs spectral‑guided invariant fusion of frozen CLIP and DINOv2 features to suppress domain‑sensitive responses while preserving semantic and structural cues, and adds a spectral decoder refinement module that normalizes low‑frequency statistics and adaptively selects frequencies to reduce residual domain bias. Experiments on synthetic‑to‑real, real‑to‑real, and adverse‑weather benchmarks show consistent improvements over prior state‑of‑the‑art methods.

By Yansong Bai, Yunzhi Zhuge, Xinzhuo Yu, Lu Zhang, Jin Zhan, Pingping Zhang, Huchuan Lu
arXiv AI
Oct 31

Localizing Concepts in Visual Autoregressive Models

The paper introduces LoCo, a model-agnostic framework that localizes where and when specific concepts appear in visual autoregressive models across Layer, Scale, and Position dimensions. It also presents LoCoBench, a dataset covering ten categories, and demonstrates the method on models such as Infinity, HunyuanImage-3.0, and InfinityStar. Using these localizations, the authors apply concept erasure, model personalization, and adversarial concept injection, achieving state‑of‑the‑art performance with reduced computational cost.

By Nanxiang Jiang, Yang Liu, Yuanhao Wang, Min Xu
arXiv AI
Oct 31

How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates

The paper introduces MSafeTemp, a large multilingual safety alignment corpus covering 60 languages, created via a template-based augmentation pipeline that extracts safe templates from harmful prompts, translates them, and fills slots with culturally authentic entities. The authors demonstrate that evaluating safety solely in English can misjudge multilingual performance, as attacks that fail in English may succeed when translated, and culturally localized variants reveal additional failures. They also show that augmenting safety training data with these templates improves safety performance beyond English.

By Max Neuwinger, Hanna Yukhymenko, Imanol Schlag
arXiv AI
Oct 31

Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble

The paper introduces Prompts to Proxies (P2P), a two‑stage inference‑time system that uses structured attribute‑based prompting and entropy‑guided sampling to build a diverse proxy pool, followed by L1‑regularized regression to create a compact weighted ensemble that matches aggregate survey responses. P2P achieves low mean squared error on American Trends Panel data and competitive performance on World Values Survey cross‑locale experiments, outperforming baseline prompting and demographic‑conditioning methods while using minimal training data. The authors argue that this preference reconstruction approach offers a lightweight, externally verifiable alternative for aligning large language model proxies with target human populations.

By Bingchen Wang, Zi-Yu Khoo, Jingtan Wang
arXiv AI
Oct 31

RePolicy: Learning to Select and Apply Safety Policies for Agent Safeguards

RePolicy is a safeguard that learns to both select the appropriate safety policy from a library and assess agent behavior against it. It processes an agent trajectory and a set of candidate policies, chooses the best-matching policy, applies its requirements, and predicts safety. The authors created PolicyTraj-20K, a dataset of over 20,000 annotated trajectories, and trained RePolicy with supervised and reinforcement learning, achieving top performance on six benchmarks and 19 baselines.

By Houcheng Jiang, Boxuan Zhang, Qiyong Zhong, Junfeng Fang, Xiang Wang, Xiangnan He
arXiv AI
Oct 31

SemantiScene-Edit: Expert-Synthesized Supervision and Dual-Path Source Conditioning for Instruction-Based Video Editing

SemantiScene-Edit is a video editing system that uses a multimodal encoder and a pretrained video diffusion transformer to follow natural‑language instructions while preserving the source video’s appearance and temporal structure. The encoder processes source frames and the instruction together, sending semantic queries to the generator via a trainable connector and cross‑attention. A dual‑path visual route concatenates source‑video latents with noisy target latents to provide dense spatial and temporal information, and the model is trained with flow matching and a three‑stage curriculum on over 1.2 million instruction‑conditioned records, with a 100‑case benchmark for evaluation.

By Xinyu Chen, Song Wu, Dingbao Shao, Jiahang Li, Jiang Lin, Yuhang Liu, Duo Li, Jiaxin Hu, Shengrong Gu, Liang Li, Qian Wang, Junlan Feng, Ying Tai, Jian Yang, Zili Yi
arXiv AI
Oct 31

Reliability of LLM Judges for Evaluating Entity Alignment

The paper investigates how reliable large language models (LLMs) are when used as judges to evaluate entity alignment (EA) systems, which match equivalent entities across knowledge graphs. Through systematic benchmarking across three frontier LLMs, three datasets, and four EA systems, the study identifies a failure mode called anchor bias, where judges invert discrimination when the system’s decision label is visible. The authors show that a label‑free evaluation protocol restores high discrimination accuracy, while visible labels cause significant score shifts, especially in stronger models—a phenomenon they term the frontier model paradox. They also provide a new biomedical EA benchmark (MeSH–SNOMED CT) and an open‑source framework for auditing LLM judge reliability.

By Vaibhava Lakshmi Ravideshik, Mayank Kejriwal
arXiv AI
Oct 31

Joint Detection of Fraud and Concept Drift inOnline Conversations with LLM-Assisted Judgment

The paper presents a two‑stage framework for detecting fraud in online conversations. First, an ensemble classifier flags suspicious dialogues, then a One Class Drift Detector isolates concept drift within those flagged exchanges. A large language model evaluates whether detected drift signals fraudulent manipulation or a legitimate topic shift, improving accuracy and interpretability compared to a Dual LLM baseline.

By Ali \c{S}enol, Garima Agrawal, Huan Liu
arXiv AI
Oct 31

MedGemma vs GPT-4: Open-Source and Proprietary Zero-shot Medical Disease Classification from Images

The paper compares the open‑source MedGemma-4b-it model, fine‑tuned with LoRA, to the proprietary GPT‑4 on a six‑disease medical image classification task. MedGemma achieved a mean test accuracy of 80.37%, outperforming GPT‑4’s 69.56%, and showed higher sensitivity for critical conditions such as cancer and pneumonia. Detailed confusion matrices and classification reports illustrate the performance differences across all disease categories.

By Md. Sazzadul Islam Prottasha, Nabil Walid Rafi
arXiv AI
Oct 31

Explainability-Aware Evaluation of Transfer Learning Models for IoT DDoS Detection Under Resource Constraints

The paper evaluates seven pre‑trained CNN architectures for detecting IoT DDoS attacks under resource constraints, using the CICDDoS2019 dataset and image‑based traffic representation. It combines performance metrics, reliability statistics, latency and training cost, and interpretability methods (Grad‑CAM and SHAP) to assess each model. DenseNet169 and MobileNetV3 emerge as top performers, offering strong reliability, interpretability, and a favorable latency‑accuracy trade‑off for fog‑level deployment.

By Nelly Elsayed
arXiv AI
Oct 31

Validity-Calibrated Reasoning Distillation

The paper introduces Validity-Calibrated Reasoning Distillation, a new framework for transferring multi-step reasoning from large language models to smaller ones. Unlike prior methods that rely on static teacher‑student hierarchies and trajectory imitation, this approach treats distillation as a local learning‑signal allocation problem. By comparing the teacher’s and student’s next‑step actions under the same prefix and modulating updates based on their relative local validity, the method provides dynamic, context‑dependent supervision that preserves structural guidance while adapting to local reasoning quality. Experiments on math, code, and instruction‑following benchmarks show that this strategy outperforms strong distillation baselines.

By Khouloud Saadi, Di Wang
arXiv AI
Oct 31

Iterative Calibration and Learning: Mitigating Teacher Bias in LLM Distillation without Target-Domain Reward Feedback

The paper introduces Iterative Calibration and Learning (IterCaL), a method for large language model distillation that mitigates teacher bias when target-domain reward feedback is unavailable. IterCaL alternates between calibrating the source-reward teacher via token-level branching and training the student on target prompts, achieving polynomial convergence to an oracle student. The authors prove that regularized direct teacher matching cannot reach the oracle, whereas IterCaL converges to it using only source rewards.

By Haichen Hu, Yuheng Zhang, David Simchi-Levi
arXiv AI
Oct 31

Algorithmic Bias from the Perspectives of Healthcare Professionals

The paper examines algorithmic bias in machine learning and AI used in healthcare information systems. Using survey and interview data from healthcare professionals in varied roles, it explores how perceptions of bias, fairness, and acceptance differ by job function and algorithm characteristics. The findings highlight the need for further research to ensure fairness and equality in healthcare AI applications.

By Jennifer Xu, Tamara Babaian
arXiv AI
Oct 31

From Geometry to Behavior: How Instruction Exposure Unlocks Latent Rhetorical Directions in LLMs

The paper investigates whether language models possess latent rhetorical abilities, specifically analogy reasoning, before instruction tuning. By extracting an "Analogy Vector" from the residual stream, the authors demonstrate that this vector can steer explanation style across several instruction-tuned models, improving performance metrics. Their findings reveal a continuous gradient of behavioral accessibility tied to the amount of instruction exposure during training, challenging the binary base‑vs‑instruct distinction and suggesting that instruction exposure shapes rhetorical knowledge in activation space.

By Shengjie Xu
arXiv AI
Oct 31

The Indian Financial Automated Advice Bias (IFAAB) Dataset : Auditing Socioeconomic and Demographic Disparities in LLM-Generated Financial Advice in India

The Indian Financial Automated Advice Bias (IFAAB) dataset presents 31,734 GPT‑4o‑generated financial advice responses for synthetic Indian personas that vary by gender, caste, education, employment, residence, age, and district. The audit reveals significant disparities: male‑labelled profiles receive 60.1 % more initial resources and 46.8 % more post‑advice resources than female‑labelled ones, while caste‑based gaps persist at 49.4 % for post‑advice resources. Textual analysis shows low semantic similarity between gendered responses, and qualitative findings indicate women are steered toward cautious saving and basic literacy, whereas men receive growth‑oriented advice.

By Kriti Mahajan
arXiv AI
Oct 31

Aligning Misaligned Benchmarks for Single-View Novel View Synthesis

The paper addresses the lack of standardized evaluation for single‑view novel view synthesis by highlighting the importance of aligning a method’s reference frame with the ground‑truth camera pose. It proposes using traditional 2D‑to‑2D and 2D‑to‑3D image matching to align 3D reconstructions and restricts evaluation to co‑visible regions where a correct ground truth exists. Experiments show that with proper alignment, 3D Gaussian‑based methods reach state‑of‑the‑art performance, and a simple baseline using a video generator and geometry estimator matches specialized novel‑view synthesis models without task‑specific training.

By Yang-Che Tseng, Erqun Dong, Jeong-gi Kwak, Gopal Sharma, Shakiba Kheradmand, Stephan Garbin, Hossam Isack, Kwang Moo Yi
arXiv AI
Oct 31

Membership Inference Attacks on Recommender System: A Survey

The article surveys membership inference attacks (MIAs) on recommender systems (RecSys), highlighting how these attacks can reveal whether a user’s interaction record was used in training and thus pose privacy risks. It reviews recent advances, design principles, challenges, and defense strategies specific to RecSys, noting that traditional MIAs are ill‑suited due to unseen posterior probabilities. The survey presents a unified taxonomy of RecSys MIAs, discusses their pros and cons, and outlines future research directions.

By Jiajie He, Xintong Chen, Xinyang Fang, Min-Chun Chen, Yuechun Gu, Wenjin Liu, Keke Chen