AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Machine Learning
4d ago

Temporal speckle enables probabilistic weights for uncertainty-aware photonic AI

The paper demonstrates that the temporal speckle of broadband light can encode probabilistic weights for Bayesian neural networks, allowing a photonic processor to perform probabilistic convolutions every 37.5 ps. By programming mean and variance into independently amplitude‑bandwidth‑controlled spectral channels, the system delivers uncorrelated samples without heavy random‑number generation. The resulting ultrafast photonic AI classifies blood‑cell images while detecting out‑of‑distribution samples and distinguishing epistemic novelty from aleatoric ambiguity on standard uncertainty benchmarks.

By F. Br\"uckerhoff-Pl\"uckelmann, H. Borras, S. U. Hulyal, L. Meyer, X. Ji, J. Hu, J. Sun, B. Klein, F. Ebert, J. Dijkstra, L. McRae, P. Schmidt, T. J. Kippenberg, H. Fr\"oning, W. Pernice
arXiv Machine Learning
4d ago

Less Loss, Fixed Weights: Logit Corrections for Forgetting

The paper investigates whether a language model’s forgetting—worsening performance on standard web text after fine‑tuning or further pretraining—can be mitigated by simple, fixed adjustments to its final token predictions. By fitting a vocabulary‑wide bias (a token‑specific bonus or penalty) and optionally a temperature scaling, the authors evaluate recovery across 14 base‑to‑derived model pairs ranging from 1B to 7B parameters. Results show modest gains: up to 10% recovery with bias alone and up to 36% when combined with temperature, though models further pretrained on math lose the most and recover little; even in a controlled simulation with random damage, the bias‑plus‑temperature fix recovers 29%.

By Hanyu Yang
arXiv Machine Learning
4d ago

PrivacyFlow: Privacy in AI Agents through Information Flow Analysis and Consequence-Aware Risk Management

PrivacyFlow introduces a new benchmark, PrivacyBench, and a defense system that separates privacy planning from task execution in autonomous LLM agents. The system traces data provenance, models recipient knowledge, and simulates inference attacks to weigh disclosure risk against task utility, allowing it to minimize or block data sharing. Experiments show that PrivacyFlow reduces privacy failures and increases completed task sequences by 22–55 percentage points, while revealing a tradeoff between privacy, utility, and user autonomy.

By Robin Staab, Tom Wollschl\"ager, Vahab Mirrokni, Reza Shokri
arXiv Machine Learning
4d ago

Model-Agnostic Search for New Physics in the Presence of Imperfect Background by Smearing the Data

The paper introduces MASSIB, a model‑agnostic search technique for new physics that learns a density ratio between data and a misspecified background distribution. By smearing the density ratio, the method defines a signal region with high signal concentration while mitigating bias in background estimation. Applied to simulated LHC events with realistic background mismodeling, MASSIB consistently identifies data‑driven signal regions and maintains strong statistical power without relying on strong assumptions about signal or background.

By Soheun Yi, John Alison, Mikael Kuusela
arXiv Machine Learning
4d ago

Federated learning for unpaired multimodal data

The paper introduces a federated learning framework for training multimodal foundation models on unpaired, fragmented data distributed across private nodes. It uses a small public anchor set and Gram matrices to align disjoint private manifolds via centered kernel alignment, avoiding transmission of private samples. Additional techniques include subspace‑stabilized fine‑tuning to handle large transformers and precision‑weighted averaging to downweight uncertain nodes.

By Anders Eklund
arXiv Computation and Language
4d ago

How Much Can Tool Calls Reveal About You? Privacy Leakage from Tool-Call Traces

The paper investigates how tool-call traces from large language model agents can reveal user attributes. Using the PrivacyTrace benchmark, the authors show that an attacker can infer user profiles with a 26.25% success rate from tool-call data, outperforming prior guessing methods. The study highlights that tool arguments and results are key evidence, and linking traces across sessions increases profiling risk.

By Donghang Duan, Qingyu Liu, Yifei Liu, Na Zou, Xia Hu, Dongrui Liu
arXiv Machine Learning
4d ago

A Cross-Layered Multi-Drone Coordination for Medical Supply Delivery during Disaster Response Management

The paper introduces CEDA, a Centralized Training with Decentralized Execution algorithm that uses a Double Deep Q-Network and a QMIX-style mixing network to coordinate multiple drones for medical supply delivery during disaster response. CEDA incorporates a Priority-Preserving Fair Scheduling strategy that encodes triage priorities, fairness, and anti-starvation rewards, enabling drones to jointly optimize routing, coordination, and energy-efficient navigation under dynamic hazards. In simulations and PX4 SITL validation, CEDA achieves over 95% patient delivery, reduces obstacle collisions by more than 95%, and maintains high triage efficiency and fairness across patient classes.

By Aneesh Calyam, Subrahmanya Chandra Bhamidipati, Zack Murry, Sharan Srinivas
arXiv Computation and Language
4d ago

Qistas: A Pre-Registered, Portable Protocol for Fairness Evaluation of Large Language Models

Qistas is a new protocol designed to evaluate how large language models treat the social, confessional, and regional constituencies of a society in a portable, pre‑registered manner. The pilot implementation focuses on Iraq and measures procedural symmetry, differential refusal, stereotype reliance, and terminological framing using paired statistical tests and a power analysis that specifies detectable effect sizes before data collection. The authors release an authoring apparatus—including a taxonomy, checklists, audit logs, and a human review notebook—and report a field test that uncovered several defect patterns common to counterfactual fairness benchmarks.

By Nawar S. Alseelawi, Mustafa S. Aljumaily
arXiv Computation and Language
4d ago

Beyond Response Scoring: An AI-aware Cognitive Supervision Framework for Automated Mathematical Short Answer Grading

The paper introduces AICG, a two‑branch framework that enhances automated short answer grading by combining RoBERTa‑based answer scoring with a cognitive supervision branch that predicts skill mastery using a Q‑matrix and G‑DINA model. It also incorporates a generative‑AI detector to weight responses based on the likelihood of AI assistance, thereby reducing bias. Experiments on over 49,000 instructor‑graded responses show that AICG cuts mean absolute error by 50.3% and raises quadratic weighted kappa from 0.798 to 0.927 compared to baseline RoBERTa regression.

By Shengran Guo, Min Chi, Feifei Qiao, Varad Patwardhan
arXiv Computation and Language
4d ago

From Transcription to Reformulation: Defining and Benchmarking Grounded Generative Speech Recognition

The paper introduces Grounded Generative Speech Recognition (GSR), redefining speech recognition as generating semantically grounded text rather than a single transcript. It defines a grounded reformulation space and a four-level hierarchy of reformulation capabilities, and presents GSR-Bench, a comprehensive benchmark covering diverse scenarios such as meeting transcription, cross-dialect translation, and robustness to silence or noise. The authors also develop a multi-dimensional evaluation framework that combines speech grounding assessment with LLM-based semantic judgment, validated by human preference studies, and analyze trade-offs and failure modes across various ASR architectures.

By Qing Wang, Qihan Hu, Hang Lv, Yan Yang, Boqing Liu, Xuechao Zhu, Zehan Li, Hongjie Chen, Jian Kang, Jie Li, Yongxiang Li, Lei Xie
arXiv Computation and Language
4d ago

KEEPER: Learning Control-Sufficient Textual Memory for Long-Horizon LLM Agents

The paper introduces KEEPER, a framework for learning control‑sufficient textual memory in long‑horizon large language model agents. It examines four auxiliary objectives that balance information content and update sensitivity, finding that reward‑related prediction yields the best task performance before and after offline reinforcement learning. A claim‑level analysis shows that incorrect claims can persist in memory even when corrective observations are made, highlighting the need to assess error persistence and valid‑information retention alongside task outcomes.

By Jinkyeong Choi, Chaebin Jeong, Junsung Jeon, Donghyeon Park
arXiv Computation and Language
4d ago

When Steering Shifts Discourse: Identifying and Intervening on Discourse-Level Changes due to Activation Steering in LLMs

The paper investigates how activation steering—adding a vector to a large language model’s residual-stream activations—alters discourse-level aspects of generated text, such as formality, evidence usage, and ambiguity. It demonstrates that these unintended changes can be largely mitigated by applying a second, complementary steering vector that targets the affected discourse dimension, restoring about 78% of the original discourse characteristics while maintaining the intended behavior in roughly 93% of outputs.

By Shivam Adarsh, Hetianyu Huang, Lau Safin, Maria Maistro, Christina Lioma
arXiv Computation and Language
4d ago

Nedimo: An Evidence-Grounded Multilingual Pipeline for AI-Generated Sleep Stories

The paper presents Nedimo, a multilingual pipeline that generates and narrates low‑arousal sleep stories using a combination of deterministic planning and large language model generation. It details a structured workflow that includes content planning, script shaping, error‑aware retries, and rigorous loudness normalization, while also recording user interaction metrics as behavioral proxies. The authors provide an evidence‑to‑requirement mapping, a guardrail architecture with explicit acceptance thresholds, and a prospective protocol for evaluating sleep‑onset effects, emphasizing that no clinical efficacy is claimed and that empirical testing is needed.

By Georgios Makridis, Stefanos Kampouris
arXiv Computation and Language
4d ago

Safin-1.5: Safety-Native Architecture and Verifiable Reasoning for Recursive Safety Improvement

Safin-1.5 is a large language model that incorporates a safety‑native architecture and a recursive safety improvement loop to enhance its own safety while maintaining utility. The architecture uses persistent safety states and hazardous‑knowledge disentanglement to isolate and remove risky knowledge with minimal impact on general capabilities. An inference‑time harness extends the model’s abilities to formal reasoning by combining language understanding with symbolic inference and verification, enabling the construction of checkable proofs.

By Jingnan Zheng, Cheng Jin, Ming Zhang, Huan Sun, Zhongtian Ma, Kaisen Yang, Ziyuan Zhou, Tong Wu, Dongcheng Zhang, Yi Zhang, Xiao Wang, Han Qi, Shu Yu, Ermo Hua, Zongji Yu, Chaozhuo Li, Xiaoyu Wen, Zhekai Chen, Jiawei Zhou, Sirui Chen, Ning Ding, An Zhang, Tat-Seng Chua, Xiangnan He, Xia Hu, Bowen Zhou, Qiaosheng Zhang, Youbang Sun, Xiang Wang, Zhongyu Huang, Chaochao Lu
arXiv Computation and Language
4d ago

SciGraphSelect: Auditable Scientific Data Selection through Knowledge--Reasoning Graphs

SciGraphSelect is a framework that makes scientific data selection auditable by using a typed knowledge–reasoning graph. It organizes domain knowledge, reusable reasoning operations, and reference‑attested relations, then scores candidate documents based on concept coverage, relation and path co‑support, and directional alignment. The approach preserves rich relational structure on the reference side while treating candidate‑side evidence conservatively, thereby exposing the evidence behind each selection.

By MiHyeon Kim, Ji-Eun Han, Eunkyeong Lee, Jeehyun Lim, Hwijung Ryu
arXiv Computation and Language
4d ago

Beyond the Black Box: Integrating Lexical and Semantic Methods in Quantitative Discourse Analysis with BERTopic

The paper introduces a transparent, hybrid framework for Quantitative Discourse Analysis that blends lexical and semantic techniques. Using custom Python pipelines with NLTK, spaCy, and Sentence Transformers, the authors fine‑tune preprocessing, lemmatisation, and embedding generation, then apply an iterative BERTopic workflow involving UMAP, HDBSCAN, and c‑TF‑IDF to produce coherent topics. The approach is illustrated through a historical political discourse case study, emphasizing code‑level transparency, researcher control, and methodological triangulation.

By Thomas Compton
arXiv Computation and Language
4d ago

Enhancing Safety of Large Language Models via Embedding Space Separation

The paper proposes Embedding Space Separation (ES2), a fine‑tuning technique that increases the distance between harmful and harmless embeddings in large language models. ES2 uses two regularization losses—a embedding anchor loss and a KL‑divergence loss—to preserve the model’s general capabilities while enhancing safety. Experiments show that ES2 effectively defends against various attack methods with only modest performance loss on some tasks.

By Xu Zhao, Xiting Wang, Weiran Shen
arXiv Computation and Language
4d ago

SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directions

The paper "SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directions" presents a unified framework for understanding autonomous RAG systems. It formalizes agentic retrieval-generation loops as finite-horizon partially observable Markov decision processes, introduces a taxonomy and modular architecture decomposition, and critiques current evaluation practices while highlighting systemic risks such as hallucination propagation and memory poisoning. The authors also propose doctoral-scale research directions for building reliable, controllable, and scalable agentic retrieval systems.

By Saroj Mishra, Suman Niroula, Umesh Yadav, Dilip Thakur, Srijan Gyawali, Shiva Gaire
arXiv Computer Vision
4d ago

Sparse Autoencoders for Expert-Assisted Interpretation of Histopathology Foundation Model Embeddings

The paper explores using sparse autoencoders (SAEs) to decompose embeddings from histopathology foundation models into interpretable latent features. By training vanilla and Batch‑TopK SAEs on NCT‑CRC embeddings from the CONCH model (and validating with UNI2‑h), the authors show that the resulting sparse representations retain most downstream tissue‑class information while producing more coherent, label‑concentrated patch groups than raw embeddings or PCA components. Pathology experts helped develop a whole‑slide‑image visualization workflow that contextualizes feature activations, underscoring that these latent features are candidate concepts needing expert assessment rather than automatically validated histopathological concepts.

By \v{S}imon \v{S}m\'ida, Wanda Benesova
arXiv Computer Vision
4d ago

RADI3N: GeometRic ImAge EDiting via ParametrIc 3D CoNtrol

RADI3N is a new framework for part‑level image editing that provides parametric 3D control. It introduces a CS‑ARAP deformation module that smooths rotation discontinuities at part boundaries, and an inversion‑free selective texture transfer mechanism that reduces positional bias and enhances semantic matching. The method achieves state‑of‑the‑art results in geometric fidelity, viewpoint consistency, and controllability.

By Abdul Wasi, Akhil Gorugantu, Mahesh Bhosale, Anjan Dutta, Junsong Yuan, Sauradip Nag, David Doermann