← Back to all news

AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv AI
Oct 21

TxnQA: Benchmarking Analytical Reasoning and Evidence Grounding over Transaction Event Sequences

TxnQA is a new analytical QA benchmark that tests reasoning and evidence grounding over transaction event sequences. It contains 27 task families with 81 templates and a 5‑scenario fraud‑risk reasoning suite built on synthetic, privacy‑mitigated transaction data. The benchmark evaluates models using format‑following rate, conditional content accuracy, and end‑to‑end correctness, revealing distinct bottlenecks in schema compliance, semantic calculation, and evidence attribution.

By Rabimba Karanjai, Yiming Bian, Yun-Shiuan Chuang, Jenna L. Ballard, Zijie Pan, Shivani Shekhar, Yixi Li, Yirou Ge, Avinash Thangali, Richard Williamson, Linsey Pang, Prakhar Mehrotra
fine-tuningbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 21

Bridging Digital Deception and Physical Fidelity: A Spatio-Temporal Attack on Cognitive Radars

The paper introduces the Segmented Spatio-Temporal Attack (SSTA), a new adversarial method for cognitive radars that addresses two key limitations of existing attacks: low cross-architecture transferability and sensitivity to physical channel distortions. SSTA uses a global‑local feature decoupling segmentation to break the dependence on long‑range sequence coherence, and incorporates differentiable spatio‑temporal operators into an Expectation over Transformation strategy to model physical distortions. Experiments on nine diverse models (CNNs, RNNs, Transformers) show a 10.36% improvement in average black‑box attack success rate, and simulations confirm that the crafted waveforms survive extreme distortions while maintaining acceptable signal quality.

By Tongrun Xing, Yongjun Dai, Xuan Liao, Zhiwen Zhang, Tong Zhou, Longlong Zhang, Tian Jiang
llmscomputer-visionbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 21

Wasserstein Robust Classification Error: Surrogate Loss Misalignment and Distributional Attacks

The paper investigates Wasserstein distributionally robust optimization (WDRO) for classification, showing that ordering hypotheses by WDRO risk aligns with robust accuracy only for a specific class of loss functions. It demonstrates that common losses like cross‑entropy can misalign robust accuracy and WDRO risk, and introduces Wasserstein distributional attacks (WDA) with a support‑search algorithm to directly evaluate robust accuracy. Experiments on CIFAR‑10, CIFAR‑100, and ImageNet reveal that WDA achieves the lowest adversarial accuracy in most settings.

By Bach C. Le, Tung V. Dao, Binh T. Nguyen, Hong T. M. Chu
safety
More like this →
arXiv Computation and Language
Oct 21

Speak vs. Know: Balancing Behavior and Knowledge Distributions of Large Language Models

The paper investigates the mismatch between large language models’ verbalized (knowledge) distributions and their actual output (behavior) distributions when predicting group opinions. It shows that for 37% of World Values Survey questions, the most likely answer under the verbalized distribution differs from that under the log‑probability distribution, indicating a disconnect between what the model knows and what it outputs. To reconcile this, the authors propose a learnable combination of the two distributions that preserves the knowledge shape while aligning with the behavior’s most likely answer, achieving higher behavioral consistency with only a modest increase in distance to human opinion distributions.

By Yang Liu, Masahiro Kaneko, Chenhui Chu
llmssafety
More like this →
arXiv Computer Vision
Oct 21

Beyond the Frame: Redundancy, Not Trajectory, Carries the Privacy Leak in Video Unlearning

The paper investigates class unlearning for video segmentation, showing that existing leakage tests underestimate privacy risks because they evaluate single frames. By averaging predictions over two‑second clips, attackers can recover forgotten classes across multiple datasets, even when per‑frame outputs appear indistinguishable from a retrain oracle. The authors attribute this leakage to cross‑frame redundancy rather than object trajectory, and propose PACE, an unlearning method that suppresses redundancy via a permutation‑invariant pooled adversarial discriminator and a two‑sided temporal consistency objective, achieving significant reductions in attacker performance while preserving segmentation quality.

By Kavyansh Tyagi, Paolo Russo
computer-visionsafety
More like this →
arXiv Computer Vision
Oct 21

LADBench: A Benchmark for Logical Fault Detection in Images

LADBench is a new benchmark comprising over 1,000 synthetic images that contain logical anomalies across four domains—Residential, Urban, Collaborative, and Nature. It introduces a Tiered Prompting Protocol that progressively discloses information to test how much explicit help a model needs to locate and reason about these faults. Evaluation of leading foundation models shows that even the best only reaches 70.11% accuracy, indicating that implicit logical fault detection remains largely unsolved.

By Sahasra Kondapalli, Lara Radovanovic, Aadi Palnitkar, Mingyang Mao, Xiaomin Lin
llmsagentsnlpmultimodalbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 21

LightSplit: Practical Privacy-Preserving Split Learning via Orthogonal Projections (Full Version)

LightSplit introduces a lightweight orthogonal random projection at the split learning cut layer to reduce communication overhead and limit information exposure. By sending low‑dimensional projections instead of raw activations, the method preserves end‑to‑end differentiability while discarding irreversibly part of the representation, thereby mitigating reconstruction attacks. Experiments on seven benchmarks show up to 32× compression with minimal accuracy loss and strong resistance to state‑of‑the‑art inversion attacks.

By Mert Cihangiroglu, Alessandro Pegoraro, Phillip Rieger, Antonino Nocera, Ahmad-Reza Sadeghi
llmscomputer-visionfine-tuningbenchmarkssafety
More like this →