AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

Simon Willison
Oct 5

Quoting Felix Rieseberg

The article explains the transition from the original Cowork system, which ran model inference and tool calls in a local Anthropic VM, to a new cloud‑based version that hosts both inference and the VM. The new design isolates each session in its own sandbox, eliminates local resource drain, and allows file access through the desktop app, addressing concerns about battery usage, performance, and continuity when a laptop is closed.

Hugging Face Trending Papers
Oct 5

SPDAlign: Interpretable Riemannian Alignment for EEG Forward Modeling Shifts

SPDAlign is an interpretable framework that aligns domain-specific EEG data by applying linear transformations on the Symmetric Positive Definite manifold, correcting mean differences and global rotations across domains using Wasserstein Procrustes. It leverages a generative model of EEG to recover distribution shifts caused by factors like head geometry, and demonstrates competitive performance on extensive public EEG datasets. The method is globally linear, enabling identification of frequency ranges, spatial patterns, and cross-subject variability without requiring labeled calibration data.

Hugging Face Trending Papers
Oct 5

MoCAR: Motion-code Coordinate-aware AutoRegression for Continuous Trajectory Forecasting

MoCAR (Motion-code Coordinate-aware AutoRegression) is a decoder‑only framework that treats continuous trajectory forecasting as next‑code prediction in a coordinate‑aware latent space. It learns a continuous motion‑code space from endpoint‑normalized trajectory segments, using historical codes as a teacher‑forced prefix and generating future codes autoregressively while updating local scene context. The method achieves top‑tier performance on Argoverse benchmarks, transfers well from AV2 to AV1 in zero‑shot evaluation, and improves on turn‑heavy scenarios without requiring trajectory‑space re‑tokenization or proposal‑and‑refinement pipelines.

Hugging Face Trending Papers
Oct 5

Benchmarking CLIP for Zero-Shot Face and Periocular Gender Estimation

The paper evaluates CLIP’s zero‑shot gender estimation on full‑face and periocular images from the Adience dataset. Using image‑text similarity with male/female prompts, CLIP achieves 95.54% accuracy on full faces without task‑specific training. Periocular predictions are initially biased toward males, but threshold alignment improves accuracy to 85.29%, and a linear SVM on CLIP features yields a modest 86.17% accuracy, slightly better than prior Adience results.

Hugging Face Trending Papers
Oct 5

TrustMI: Causally controlling how assistants trust their users

The paper introduces TrustMI, a method to causally control how large language model assistants decide to trust their users. By creating 2,000 contrastive conversations that vary in ability, benevolence, and integrity, the authors learn steering matrices that adjust trust decisions along linear directions in model activations while keeping the model parameters frozen. Experiments across six instruction‑tuned models show that these steering changes reliably alter trust decisions and affect safety‑related behaviors such as compliance with harmful requests, prompt injections, and insider threats.

Hugging Face Trending Papers
Oct 5

Certification of Real Images through Calibrated Content Authentication

The paper examines the reliability of deepfake detectors, noting a decline in accuracy from 99.5% to 76% over four years and a drastic drop below 2% when adversarial perturbations are applied. It argues that content alone cannot determine provenance because generators can perfectly reproduce authentic images, so a new detection approach is proposed that certifies authenticity only if a generator cannot faithfully reconstruct the content. The authors demonstrate that their calibrated detector can limit false certifications to 1% while maintaining robustness against bounded‑perturbation attacks, though it struggles with arbitrary transformations.

Hugging Face Trending Papers
Oct 5

Online AutoML: Evaluating Poisoning Attacks on Adversarial Training Defense Strategy in IoT Networks

The paper investigates how poisoning attacks—specifically label flip and noise injection—affect online AutoML learners in IoT networks and evaluates the effectiveness of adversarial training (AT) as a defense. Experiments on streaming-capable models (Hoeffding Tree, Leveraging Bagging, Adaptive Random Forest, Hoeffding Adaptive Tree, and Streaming Random Patches) show that AT-SRP achieves the highest F1-score (0.904) against label flip attacks, while AT-LB achieves the highest F1-score (0.933) against noise injection. Drift detection methods were also employed for rolling accuracy and prequential evaluation.

Hugging Face Trending Papers
Oct 5

Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating

The paper introduces U-LoRA, an adaptive version of Low‑Rank Adaptation that uses conditioned gating to generate token‑specific utilization coefficients for the low‑rank subspace. By coordinating these coefficients with sequence‑level context and applying a bias‑corrected EMA prior, U‑LoRA achieves more consistent adaptive patterns and reduces noise from batch fluctuations. Experiments on mathematical reasoning and natural language understanding tasks show that U‑LoRA matches or surpasses strong LoRA baselines while keeping the same parameter budget.

arXiv Computer Vision
Oct 5

The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animation

The paper introduces a geometric measure of coarticulation for speech‑driven 3D facial animation, comparing lip‑path length to the shortest route through vowel, consonant, and vowel positions. Using only forced alignment, the measure evaluates four state‑of‑the‑art animation methods, revealing that all produce flatter lip trajectories than captured speech and that some methods lose 15–60% of the fast articulatory component. A pre‑registered viewer study confirms that damping real motion lowers perceived quality while exaggeration is not penalized, and viewers prefer real speech in 73.4% of sentence comparisons.

By Danzel Serrano, Przemyslaw Musialski
arXiv Machine Learning
Oct 5

Architecture-Dependent Fusion Pathways in MLLMs

The paper investigates how visual and textual information are fused in Multimodal Large Language Models (MLLMs). By analyzing concatenation and native multimodal architectures through alignment decoupling, attention routing, entropy, intrinsic dimensionality, and causal interventions, the authors uncover two distinct fusion pathways: concatenation models use a text‑first, vision‑later strategy, while native models integrate vision and text earlier and reorganize feature spaces. The study also employs visual CKA to test the Platonic Representation Hypothesis, offering a mechanistic view of multimodal fusion and informing architecture‑aware diagnostics.

By Hebao Zhu, Dongxia Wu
arXiv Computation and Language
Oct 5

Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation

The study explores how the language used for reasoning affects retrieval‑augmented generation (RAG) in a monolingual German setting. Using a German RAG question‑answering testbed based on the tabletop game The Dark Eye, the authors show that aligning the reasoning language with the query and retrieved documents improves performance, with German reasoning outperforming French reasoning. However, German reasoning still does not surpass the model’s native English reasoning, indicating that native multilingual reasoning is necessary for optimal results.

By Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense