AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

10,474 stories · RSS feed

arXiv AI

Jun 30

Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.

By Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella Biderman

llms benchmarks safety

arXiv Machine Learning

Jun 30

Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning

arXiv:2605. 29032v2 Announce Type: replace Abstract: Model-based reinforcement learning (MBRL) agents typically learn world models by minimizing predictive loss.

By Christoph Dann, Yishay Mansour, Mehryar Mohri

agents reinforcement-learning safety

arXiv AI

Jun 30

Hierarchical Reinforcement Learning in StarCraft Micromanagement with Influence Maps and Cluster-based Scripts

arXiv:2606. 30092v1 Announce Type: new Abstract: Real-time strategy (RTS) games present significant AI challenges, characterized by expansive state-action spaces arising from multi-unit coordination in continuous battlefields, and sparse delayed rewards stemming from final win/lose signals.

By Chunhui Bai, Changhe Li, Dequan Li, Xinye Cai, Shengxiang Yang

reinforcement-learning safety

arXiv AI

Jun 30

The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governance

arXiv:2606. 28710v1 Announce Type: new Abstract: We ask under what conditions an agent with a harm-minimizing policy can displace an approval-seeking (RLHF) agent in a competitive market, and when that policy is sufficient to prevent community harm.

By Darrell Lewis-Sandy

agents reinforcement-learning safety

arXiv Machine Learning

Jun 30

Constrained Policy Optimization with Cantelli-Bounded Value-at-Risk

arXiv:2601. 22993v4 Announce Type: replace Abstract: We introduce Canary, a risk-averse method designed to optimize Value-at-Risk (VaR) constrained reinforcement learning (RL) problems.

By Rohan Tangri, Jan-Peter Calliess

reinforcement-learning benchmarks safety

arXiv AI

Jun 30

RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation

arXiv:2606. 28410v1 Announce Type: cross Abstract: Open-vocabulary semantic segmentation (OVSS) enables text-guided segmentation of unseen objects, breaking fixed-class limitations to achieve open-world understanding.

By Shanwen Wang, Xin Sun, Sirui Wang, Xiao Xiang Zhu

llms computer-vision benchmarks safety

arXiv Machine Learning

Jun 30

Singular Learning and Occam's Razor in Deep Monomial Networks

arXiv:2606. 28464v1 Announce Type: new Abstract: In the optimization of neural networks, gradient dynamics are influenced by critical points that arise from the model's architecture.

By Kathl\'en Kohn, Giovanni Luca Marchetti, Farhan Shabir, Vahid Shahverdi, Weisheng Wang

safety

arXiv AI

Jun 30

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

arXiv:2606. 29357v1 Announce Type: cross Abstract: Vision-language tracking guided by natural language specifications leverages high-level semantic cues of target objects to substantially boost tracking accuracy and robustness.

By Xiao Wang, Liye Jin, Dan Xu, Yuehang Li, Lan Chen, Yaowei Wang, Yonghong Tian, Jin Tang

llms multimodal benchmarks safety

arXiv AI

Jun 30

Evidence-Based Text-Conditioned 3D CT Synthesis for Ovarian Cancer

arXiv:2606. 28980v1 Announce Type: cross Abstract: Ovarian cancer is frequently diagnosed at an advanced stage, making preoperative contrast-enhanced computed tomography (CT) central to staging and surgical planning; yet the scarcity of annotated imaging data, compounded by privacy regulations, limits the development of generalizable computational models in this domain.

By Francesca Pia Panaccione, Eugenio Lomurno, Francesca Fati, Carlotta Pecchiari, Marina Rosanu, Luigi De Vitis, Lucia Ribero, Gabriella Schivardi, Giovanni Damiano Aletti, Nicoletta Colombo, Maria Francesca Spadea, Francesco Multinu, Matteo Matteucci, Elena De Momi

diffusion fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 30

R-DMesh: Video-Guided 3D Animation via Rectified Dynamic Mesh Flow

arXiv:2605. 13838v3 Announce Type: replace-cross Abstract: Video-guided 3D animation holds immense potential for content creation, offering intuitive and precise control over dynamic assets.

By Zijie Wu, Lixin Xu, Puhua Jiang, Sicong Liu, Chunchao Guo, Xiang Bai

llms diffusion safety

arXiv Machine Learning

Jun 30

Road Maps as Free Geometric Priors: Weather-Invariant Drone Geo-Localization with GeoFuse

arXiv:2605. 14925v2 Announce Type: replace-cross Abstract: Drone-view geo-localization aims to match a query drone image, often captured under adverse weather conditions (e.

By Yunsong Fang, Tingyu Wang, Zhedong Zheng

robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Inference-time optimization for experiment-grounded protein ensemble generation

arXiv:2602. 24007v3 Announce Type: replace-cross Abstract: Protein function relies on dynamic conformational ensembles, yet current generative models like AlphaFold3 often fail to produce ensembles that match experimental data.

By Advaith Maddipatla, Anar Rzayev, Marco Pegoraro, Martin Pacesa, Paul Schanda, Ailie Marx, Sanketh Vedula, Alex M. Bronstein

rag diffusion benchmarks safety

arXiv Machine Learning

Jun 30

LoRAShield: Data-Free Editing Alignment for Secure Personalized LoRA Sharing

arXiv:2507. 07056v2 Announce Type: replace-cross Abstract: The proliferation of Low-Rank Adaptation (LoRA) models has democratized personalized text-to-image generation, enabling users to share lightweight models (e.

By Jiahao Chen, Junhao Li, Yiming Wang, Yong Yang, Yi Jiang, Chunyi Zhou, Qingming Li, Tianyu Du, Shouling Ji

llms diffusion fine-tuning safety

arXiv Machine Learning

Jun 30

Reinforcement Learning with a Bilevel World-Model Architecture for Scan-Order Optimisation in Laser Directed Energy Deposition

arXiv:2605. 25063v2 Announce Type: replace Abstract: Scan-order design in laser directed energy deposition (LDED) is a delayed, path-dependent thermo-mechanical decision problem, because sequence quality becomes observable only after the complete deposition and cooling cycle.

By Xian Wu, Haoran Li, Yuanqi Chu, Dongbin Zhao, Bin Wang

reinforcement-learning safety

arXiv Machine Learning

Jun 30

$\mu$Flow: Leveraging Average Images for Improving Generalisation of Deepfake Faces Detectors

arXiv:2606. 30528v1 Announce Type: cross Abstract: Current generative models, including GANs and diffusion models, have reached an outstanding level of photorealism, posing significant risks to privacy and security.

By Orazio Pontorno, Mattia Litrico, Luca Guarnera, Mario Valerio Giuffrida, Sebastiano Battiato

diffusion benchmarks safety

arXiv AI

Jun 30

Modification-Considering Value Learning for Reward Hacking Mitigation in RL

arXiv:2606. 28955v1 Announce Type: cross Abstract: Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective, a failure mode known as reward hacking.

By Evgenii Opryshko, Umangi Jain, Igor Gilitschenski

agents reinforcement-learning safety

arXiv Machine Learning

Jun 30

REAR: Test-time Preference Realignment through Reward Decomposition

arXiv:2606. 30339v1 Announce Type: cross Abstract: Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task.

By Fuxiang Zhang, Pengcheng Wang, Chenran Li, Yi-Chen Li, Yuxin Chen, Lang Feng, Chenfeng Xu, Masayoshi Tomizuka, Bo An

llms safety

arXiv Machine Learning

Jun 30

A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctions

arXiv:2606. 28943v1 Announce Type: cross Abstract: Learning to bid in repeated multi-unit auctions with bandit feedback poses a fundamental challenge.

By Junhan Li, Yuxin Zhang, Haoran Wang, Minghao Chen

reinforcement-learning benchmarks safety

arXiv AI

Jun 30

High-Dimensional Concentration and Retrieval Instability in Embedding Spaces: Implications for Retrieval-Augmented Generation

arXiv:2606. 28330v1 Announce Type: cross Abstract: Embedding-based retrieval systems rely on the assumption that geometric proximity in highdimensional representation spaces reflects semantic relevance.

By Ernesto Lopez Fune (DE)

rag safety

arXiv Machine Learning

Jun 30

Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders

arXiv:2606. 29888v1 Announce Type: new Abstract: Vision-language models map images and text into a joint embedding space.

By Chungpa Lee, Jihoon Kwon, Kyle Min, Jy-yong Sohn

llms rag multimodal safety

Get the daily briefing