AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,798 stories · RSS feed

arXiv Machine Learning
Oct 8

Beyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM Yielding

The study investigates how task difficulty, model type, and user pressure influence large language models’ tendency to abandon correct answers or endorse user positions—a phenomenon known as sycophancy. Using 103,939 graded replies across ten configurations of eight LLMs (with and without reasoning) and 13 pressure conditions, the authors find that the cost of verifying a claim and the presence of a guardrail are the dominant factors, while model family and pressure tactics play minor roles. Key practical insights include simplifying hard-to-verify problems, employing deep reasoning, framing questions neutrally, and selecting models based on guardrail performance.

By Guang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian
arXiv Machine Learning
Oct 8

Covariate-dependent Joint Modeling of Multivariate Ordinal Preferences and Its Connections with Comparison Models

The paper introduces a covariate‑dependent joint modeling framework for multivariate ordinal preference data, addressing limitations of existing methods that treat attributes separately or reduce observations to pairwise comparisons. It proposes a consecutive‑ratio Markov random field model that retains the full ordinal structure and demonstrates that standard comparison models emerge as special cases under restrictive assumptions. The authors also present a maximum‑likelihood inference procedure capable of handling an intractable normalizer.

By Yujie Chen, Antik Chakraborty, Anindya Bhadra
arXiv Machine Learning
Oct 8

Beyond Explanation: Debugging Medical Imaging Models via Concept Intervention

The paper presents a plug‑and‑play framework that uses concept-based interpretation to debug medical imaging models. By aligning a single‑modality encoder with BioMedCLIP, the authors build a Concept Bottleneck Model that allows concept‑level interventions to distinguish causal from spurious features. Experiments on Mayo Clinic ultrasound and CheXpert chest X‑ray data show that these interventions can reliably diagnose model issues and even improve performance through guided fine‑tuning.

By Samrajya Thapa, Daniel J. Quest, Timothy L. Kline, Carrie L. Langstraat, Emanuel C. Trabuco, Wei Le
arXiv Machine Learning
Oct 8

HE-OFT: Privacy-Preserving One-Shot Federated Fine-Tuning under Homomorphic Encryption

HE‑OFT is a cryptographically secure one‑shot federated fine‑tuning protocol that protects privacy by exchanging only one encrypted contribution and never revealing the trained model. Each client fine‑tunes a low‑rank adapter and a classifier head on a frozen public backbone, keeps the adapter, and uploads an encrypted head displacement that the server combines using multiparty CKKS without decryption. On four text classification tasks and one vision task, HE‑OFT achieves 61–79 % accuracy, retaining 85–96 % of the accuracy of a disclosed model, while a test‑time query takes 443.1–1713.1 s on one core or 56.1–255.1 s on a GPU, with level restoration reducing traffic from 1.6 GiB to 13.5 MiB.

By Halil \.Ibrahim Kanpak, Sinem Sav, Alptekin K\"up\c{c}\"u
arXiv Machine Learning
Oct 8

The Winner's Curse in LLM Self-Improvement Loops: Selection Noise, Lock-in, and Acceptance Rules

The paper investigates how self‑improving large language models (LLMs) can suffer from the winner’s curse when selecting better candidates based on noisy evaluation sets. By modeling correlated errors and empirically testing Qwen models that rewrite their own instructions, the authors find that most proposals after the first are harmful and that the reported gains on the selection set often overstate true held‑out performance. The study shows that larger selection sets can improve held‑out gains on some tasks (e.g., TREC) but not others (e.g., GSM8K), and that common acceptance rules do not outperform simple greedy selection across entire runs.

By Litao Hu, Yutong Tang
arXiv Computer Vision
Oct 8

Certification of Real Images through Calibrated Content Authentication

The paper evaluates twenty deepfake detectors against ten recent generators, finding that accuracy has dropped from 99.5% to 76% and that adversarial perturbations can reduce detector performance to below 2%. It proposes a new detection paradigm that certifies authenticity only when a faithful reconstruction by a known generator is impossible, and shows that this calibrated detector can limit false certifications to at most 1% while remaining robust to bounded‑perturbation attacks. The study also highlights the erosion of post‑hoc verifiability, noting that newer generators can reproduce a larger fraction of previously unreplicable images.

By Sarim Hashmi, Abdelrahman Elsayed, Mohammed Talha Alam, Samuele Poppi, Nils Lukas
arXiv Machine Learning
Oct 8

BeatFlow-ECG: Rectified Flow for ECG Reconstruction from Indirect Wearable Signals

BeatFlow-ECG is a conditional rectified‑flow model that reconstructs single‑channel ECG signals from synchronized photoplethysmography (PPG) and inertial measurement unit (IMU) data. The architecture uses a convolutional encoder‑decoder with a transformer bottleneck and explicit flow‑time conditioning, incorporating motion information through IMU‑derived features, motion‑dependent loss weighting, and a curriculum learning strategy. Evaluated on PPG‑DaLiA and WESAD datasets with leave‑one‑subject‑out protocols, BeatFlow‑ECG outperforms deterministic, adversarial, and diffusion‑based baselines, achieving Pearson correlations of 0.983–0.986 and R‑peak F1 scores of 0.946–0.955, while reducing L1 error compared to Conditional DDPM‑1D.

By Mohamed Kamel, Sahar Selim, Walaa Medhat, Tamer Nadeem
arXiv Machine Learning
Oct 8

Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization

The paper introduces Counterfactual KKT (CF‑KKT), a framework that learns unknown constraints from locally optimal demonstrations without requiring known dynamics or risky exploration. CF‑KKT uses a learned differentiable dynamics model to apply KKT‑inspired optimality conditions and generates counterfactual behaviors that reveal infeasible data, enabling safer and more data‑efficient constraint recovery. Experiments on high‑dimensional robotic control tasks show that neural constraint representations learned with CF‑KKT achieve better safety and data efficiency than state‑of‑the‑art offline inverse constrained reinforcement learning baselines.

By Zhouyu Zhang, Chih-Yuan Chiu, Glen Chou
arXiv Machine Learning
Oct 8

Democratizing MoE inference on commodity GPUs with CoMoE

The paper introduces CoMoE, a communication‑efficient Mixture‑of‑Experts inference system designed for commodity GPUs. By using host‑centric routing, host‑backed token multicast, and fine‑grained token‑level aggregation, CoMoE reduces communication volume and eliminates global synchronization stalls. Experiments on RTX 5090 GPUs show up to 1.46× throughput improvement, approaching NVLink‑capable A800 performance while costing only 23.4% of the hardware.

By Ruwen Fan (Jimmy), Yuezhi Zu (Jimmy), Junru Li (Jimmy), Qingda Hu (Jimmy), Xinjun (Jimmy), Yang, Jiwu Shu, Youyou Lu
arXiv Machine Learning
Oct 8

Unpaired Canonical Correlation Analysis

Unpaired Canonical Correlation Analysis (UCCA) introduces a method for learning linear projections that maximize correlation between views without requiring paired data. By linking the Quadratic Assignment Problem to CCA, the authors derive a practical algorithm that operates solely on unpaired samples. Experiments on real-world multimodal datasets show that UCCA outperforms existing unpaired alignment baselines in recovering true correlations.

By Nir Ben-Ari, Ronen Talmon, Uri Shaham
arXiv Machine Learning
Oct 8

Distilling Graph Geometry: Knowledge Gap from GNNs to MLPs

The paper introduces G^2MLP, a graph‑free MLP trained via distillation from a GNN teacher while preserving the teacher’s graph‑induced geometry. It identifies two spectral failure modes—underfit on sparse graphs and overfit on dense graphs—caused by neglecting geometry during distillation. By using Ollivier‑Ricci curvature to guide supervision between prediction‑level and representation‑level alignment, G^2MLP improves node‑classification performance and reduces the teacher‑student rank gap across benchmarks.

By Zhewei Chen, Hao Zhu, Jiaojiao Jiang, Ahad N. Zehmakan
arXiv Machine Learning
Oct 8

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

The paper proposes that independently trained multimodal models often share a common geometric structure, allowing cross‑modal alignment without paired data. Using a Wasserstein Procrustes approach with a coarse geometric initialization, the authors align two disjoint embedding sets by estimating a single orthogonal map, achieving consistent alignment across datasets and modalities. While paired examples can further improve performance, the method already outperforms existing techniques in the few‑pair regime and enables text‑to‑image generation without paired data.

By Dominik Schnaus, Thomas Dag\`es, Daniel Cremers, Xi Wang, Phillip Isola
arXiv Machine Learning
Oct 8

emg2face: Expressive Facial Animation with High-Density Surface EMG

The paper presents emg2face, a system that uses high‑density surface electromyography (HD‑sEMG) to capture facial expressions without optical cameras, addressing issues of occlusion and privacy. It records 64 EMG channels with textile grids, synchronizes the data with video using analog audio bursts, and fits a high‑resolution parametric head model to 3D facial landmarks. A deep neural network then predicts blendshape parameters from the EMG signals, enabling real‑time facial animation on various characters.

By Ganidhu Abey, Wendy Greening, Ashika Kamboj, Leonhard Helminger, Abhijeet Ghosh, Karel Petranek, Sergio Orts Escolano, Dinesh K. Pai
arXiv Machine Learning
Oct 8

Careful Judge: Safe and Efficient Human-AI Collaborative Decision Making

The paper introduces CARE, a calibrated adaptive rectification and escalation framework that integrates AI models with human reviewers to ensure safe, human-aligned decisions. CARE continuously learns from human feedback, improving automation while reducing the need for human queries. Experiments on driving, language, and robotics datasets show a 25–81% reduction in human queries compared to baseline methods.

By Chenyu Zhang, Rachel Luo, Boyi Li, Anjali Parashar, Marco Pavone, Apoorva Sharma