AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

10,474 stories · RSS feed

arXiv Machine Learning

Jul 1

Relational and Sequential Conformal Inference for Energy Time Series over Graphs via Foundation Models

arXiv:2606. 31804v1 Announce Type: new Abstract: Accurate energy demand forecasting is essential for the reliable operation and planning of modern sustainable energy systems.

By Keivan Faghih Niresi, Alice Cicirello, Olga Fink

llms benchmarks safety

arXiv Machine Learning

Jul 1

Safe Online Learning via Smooth Safety-Structured Policy Composition

arXiv:2606. 31320v1 Announce Type: new Abstract: Safe online reinforcement learning requires policies to respect safety constraints while maintaining smooth optimization dynamics.

By Hongpeng Cao, Liqun Zhao, Yuliang Gu, Naira Hovakimyan, Lui Sha, Marco Caccamo

reinforcement-learning benchmarks safety

arXiv AI

Jul 1

Teaching LLMs String Matching, Backtracking, and Error Recovery to Deduce Bases and Truth Tables for the Combinatorially Exploding Bit Manipulation Puzzles

arXiv:2606. 23672v2 Announce Type: replace Abstract: This paper presents our algorithmic innovations for the NVIDIA Nemotron Model Reasoning Challenge, focusing on Bit Manipulation Puzzles.

By Prateek Agnihotri, Sanchit Jain, Prabhat Agnihotri, Aditya Prasad, Shubham Jain

llms agents nlp robotics fine-tuning safety

arXiv AI

Jul 1

Test-Time Verification for Text-to-SQL via Outcome Reward Models

arXiv:2606. 30851v1 Announce Type: cross Abstract: Improving the reliability of large language models (LLMs) at inference time is a central challenge in structured reasoning tasks such as Text-to-SQL.

By Mattia Tritto, Giuseppe Farano, Dario Di Palma, Gaetano Rossiello, Fedelucio Narducci, Dharmashankar Subramanian, Tommaso Di Noia

llms benchmarks safety data-engineering

arXiv Machine Learning

Jul 1

A Complete Characterization of Learnability for Adversarial Noisy Bandits

arXiv:2605. 09200v2 Announce Type: replace Abstract: We study adversarial noisy bandits given a known function class $\mathcal{F}$.

By Steve Hanneke, Kun Wang

reinforcement-learning safety

arXiv AI

Jul 1

Containment Verification: AI Safety Guarantees Independent of Alignment

arXiv:2605. 09045v2 Announce Type: replace Abstract: Agentic frameworks are the software layer through which AI agents act in the world.

By Royce Moon, Lav R. Varshney

llms agents safety

arXiv AI

Jul 1

IPO Finance Agent: Benchmark of LLM Financial Analysts Beyond Finance Agent v2, with Automated Rubric Generation, on the SpaceX (SPCX) IPO

arXiv:2606. 23032v3 Announce Type: replace Abstract: Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating both Anthropic Claude and OpenAI ChatGPT frontier language models on financial tasks.

By Mostapha Benhenda

llms agents benchmarks safety

arXiv AI

Jul 1

Deductive Logic in Language Models: Horizontal vs Vertical Reasoning

arXiv:2510. 09340v2 Announce Type: replace Abstract: Recent language models exhibit significant logical reasoning abilities, yet the mechanisms supporting deductive inference remain poorly understood.

By Davide Maltoni, Matteo Ferrara

llms safety

arXiv Machine Learning

Jul 1

Personalizing Marketplace Policies with Competing Objectives and Constrained Experiments: Evidence from a Job Marketplace

arXiv:2606. 30932v1 Announce Type: new Abstract: Two-sided marketplaces connect distinct user groups whose interests often conflict -- improving outcomes on one side could degrade the other side's experience.

By Yufei Wu, Zhen Yan

safety

arXiv Machine Learning

Jul 1

Coarsening Bias from Variable Discretization in Causal Functionals

arXiv:2602. 22083v2 Announce Type: replace-cross Abstract: Causal identification functionals often require integration over conditional densities of continuous variables, such as those arising in nonparametric identification theory of total and mediated causal effects in DAGs with hidden variables.

By Xiaxian Ou, Razieh Nabi

safety

arXiv AI

Jul 1

Amplifying Membership Signal Through Chained Regeneration

arXiv:2606. 31991v1 Announce Type: cross Abstract: The tendency of large generative models to memorize training data makes sample verification critical for privacy auditing and copyright enforcement.

By Wojciech {\L}apacz, Stanis{\l}aw Pawlak

llms diffusion safety

arXiv Machine Learning

Jul 1

Incentive Aware AI Regulations: A Credal Characterisation

arXiv:2603. 05175v2 Announce Type: replace Abstract: The rapid proliferation of AI applications has intensified debate on effective regulation of these black-box services.

By Anurag Singh, Julian Rodemann, Rajeev Verma, Siu Lun Chau, Krikamol Muandet

safety

arXiv AI

Jul 1

Understanding and Evaluating Claw-like Agent Security Through a Computer-Systems Lens

arXiv:2606. 30755v1 Announce Type: cross Abstract: Claw-like AI agents (e.

By Peizhi Niu, Wenjie Qu, Shangding Gu, Tianneng Shi, Yuankai Li, Ahmad Tawaha, Hend Alzahrani, Vincent Siu, Boyi Li, Chenguang Wang, Jiaheng Zhang, Basel Alomair, Ming Jin, Muhao Chen, Chi Wang, Costas Spanos, Dawn Song

llms agents benchmarks safety

arXiv Machine Learning

Jul 1

Private Rate-Constrained Optimization with Applications to Fair Learning

arXiv:2505. 22703v2 Announce Type: replace Abstract: Many problems in trustworthy ML can be expressed as constraints on prediction rates across subpopulations, including group fairness constraints (demographic parity, equalized odds, etc.

By Mohammad Yaghini, Tudor Cebere, Michael Menart, Aur\'elien Bellet, Nicolas Papernot

safety

arXiv AI

Jul 1

Improving Certified Robustness via Adversarial Distillation

arXiv:2606. 31653v1 Announce Type: cross Abstract: Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set.

By Matteo Melis, Jesus Martinez Del Rincon, Vishal Sharma

efficiency benchmarks safety

arXiv AI

Jul 1

Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models

arXiv:2606. 31338v1 Announce Type: cross Abstract: Recent music audio-language models achieve high accuracy on instrument question-answering benchmarks, but it remains unclear whether this reflects robust audio grounding or benchmark-specific shortcuts.

By Yujun Lee, Joonhyeok Shin, Hyoeun Kim, Kyuhong Shim

llms benchmarks safety

arXiv AI

Jul 1

Bridging Local Observation and Global Simulation in Closed-Loop Traffic Modeling

arXiv:2606. 31844v1 Announce Type: cross Abstract: A local-to-global context mismatch arises when autoregressive traffic simulators trained on ego-centric driving logs are deployed in globally observable closed-loop environments.

By Ziyan Wang, Tan Xiang, Peng Chen, Xintao Yan

agents safety

arXiv AI

Jul 1

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

arXiv:2606. 32032v1 Announce Type: cross Abstract: Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes.

By Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 1

A Tutorial on Autonomous Fault-Tolerant Control Using Knowledge-Grounded LLM Agents

arXiv:2606. 31635v1 Announce Type: cross Abstract: Fault recovery in process plants still relies heavily on plant operators, especially when faults fall outside predefined supervisory logic.

By Javal Vyas, Milapji Singh Gill, Artan Markaj, Felix Gehlhoff, Mehmet Mercang\"oz

llms agents safety

arXiv Machine Learning

Jul 1

Signed-Permutation Coordinate Transport for RMSNorm Transformers

arXiv:2606. 31963v1 Announce Type: new Abstract: Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, and merge alignments.

By John Sweeney

llms fine-tuning safety

Get the daily briefing