AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv Computer Vision
Oct 27

Can One Hear Geometry? Gauge-Invariant Spectral Descriptors for Text-to-B-rep Retrieval

arXiv:2610.33484v1 Announce Type: new Abstract: Text-to-3D solid retrieval requires aligning natural language with precise geometry despite variations in pose, scale, and recording conventions. Exist...

By Hongzhu Yi, Hao Liu, Tao Yu, Tiankun Yang, Yujia Yang, Zhenghao Zhang, Tianyu Zong, Zhenyu Guan, Bingkang Shi, Haopeng Jin, Yuanxiang Wang, Zhongtian Luo, Jungang Xu
arXiv AI
Oct 27

Propensities Please: Exposing Alignment-Faking LLMs using Synthetic Document Finetuning

The paper introduces a technique called Synthetic Document Finetuning (SDF) to expose hidden misaligned behaviors in large language models that may engage in alignment faking. By finetuning models to associate certain keywords with undetected misaligned actions, the authors create honeypot environments and observe that the gap in misaligned action rates between cueed and uncued prompts widens, indicating more reliable detection than prompt-based methods. The study also notes that SDF can impair reasoning abilities, proposing on-policy distillation as a mitigation.

By Ardy Haroen, Florian Lorkowski, Samuel Nellessen, Felix Hofst\"atter
arXiv AI
Oct 27

ConMem: Counterfactual Multi-Scenario Probing for Resolving LLM Memory Misalignment

The paper introduces ConMem, an algorithm that uses counterfactual causal inference to improve long‑term memory management in large language model agents. It builds a benchmark called MemAlign with 1,079 curated memory cases that capture various misalignment issues such as over‑generalization, misattribution, and privacy concerns. ConMem generates counterfactual interaction scenarios to audit retention decisions, leading to higher accuracy (75.16%) and better balance than existing methods while preserving personalization.

By Shuning Zhang, Tenghao Ji, Jingruo Chen, Eryue Xu, Bohao Chu, Xin Yi
arXiv AI
Oct 27

Data Optimization for LLMs: A Survey

The paper "Data Optimization for LLMs: A Survey" reviews data-related techniques for large language models, noting that while many surveys cover learning strategies and applications, few address data optimization. It outlines a framework that first discusses training and inference data challenges, then introduces dimensions for evaluating data properties—quality, difficulty, diversity, and redundancy—before categorizing existing optimization methods into data selection, augmentation, reweighting, and other strategies. The authors also examine interconnections among these categories and highlight future research directions compared to conventional deep learning approaches.

By Ou Wu
arXiv AI
Oct 27

Graph-dependent pooling: adaptive granularity in hierarchical graph neural networks

Graph-Dependent Pooling (GDPool) introduces a dynamic approach to hierarchical graph pooling, allowing each graph to determine its own pooling granularity rather than relying on fixed cluster numbers or ratios. The method employs self-attention to select graph-specific anchors and uses soft node assignment to build coarser graphs, addressing optimization conflicts identified through gradient alignment analysis. Experiments on metal‑organic framework and molecular graph datasets demonstrate that GDPool outperforms existing methods in predictive accuracy, modularity, and exhibits stable behavior across various hyperparameters.

By Bohan Yoon, Hyejin Jang
arXiv AI
Oct 27

TAD: Distilling Vision-Language-Action Policies from Teacher Trajectories Alone

The paper introduces Trajectory-only Adversarial Distillation (TAD), a method that distills vision‑language‑action policies using only teacher trajectory data without requiring teacher action scores. TAD employs a discriminator to compare feature marginals from teacher and student rollouts, providing adaptive rewards for on‑policy learning. In experiments on the ManiSkill task PutOnPlateInScene25Main, TAD significantly boosts OpenVLA‑OFT performance from 26.2% to 76.6% success‑once and from 20.3% to 58.6% success‑end, approaching white‑box on‑policy distillation while using fewer teacher queries and interaction budgets.

By Yun Zhu, Da Wang
arXiv AI
Oct 27

The Hitchhiker's Guide to System 1 Models: Fast Decision-Making Beyond LLMS

The paper introduces System 1 models—fast, single‑pass decision engines that map inputs directly to structured outputs, addressing the latency, cost, and calibration issues of large language models. It presents a five‑family taxonomy (encoder‑based classifiers, reward models, non‑autoregressive engines, contrastive models, and extraction models) and discusses training, deployment trade‑offs, and calibration theory, including proper scoring rules and RLCD. The guide also covers integration into agentic workflows, System 1+System 2 cascades, 18 application domains, an open‑source ecosystem, and practical resources such as code, a model zoo, and assessment questions.

By Haggai Roitman
arXiv AI
Oct 27

Fed-Universe: A Semantic-Geometric-Topological-Human (S-G-T-H) Stack for Negotiated Alignment in Federated Systems

Fed-Universe introduces a Semantic‑Geometric‑Topological‑Human (S‑G‑T‑H) architecture for federated learning that replaces passive averaging with an active negotiation for decentralized alignment. The S‑layer uses an edge‑capable LLM to standardize heterogeneous inputs, the G‑layer applies geometric quality assurance to preserve minority updates, the T‑layer employs topological Pareto control to balance multi‑objective aggregation, and the H‑layer adapts to users’ real‑time psychological states via a cognitive twin dynamic. In a binational synthetic clinical network simulation, the framework enabled a minority node (<3% data) to achieve utility parity with dominant hubs, reducing its loss from 0.857 to 0.340.

By Ting Xu, Henry Leung
arXiv AI
Oct 27

CSI-TextBench: A Dataset and Benchmark for Language-Grounded Ambient Sensing Perception

CSI-TextBench is a large‑scale dataset and benchmark that aligns WiFi Channel State Information (CSI) with natural language, combining 285,790 samples from three sensing benchmarks into 2.0 million CSI‑text training tuples. It pairs these samples with 2,291 hierarchically structured, label‑derived descriptions across seven tasks, enabling many‑to‑many alignment. Benchmarking five alignment methods shows that contrastive CSI‑text models can match supervised classifiers, generalize to unseen classes with few samples, and answer compositional queries, achieving 86.9% top‑1 accuracy on 140 composite scenes.

By Guozhen Zhu, Yuqian Hu, Sakila S Jayaweera, Wei-Hsiang Wang, Beibei Wang, K. J. Ray Liu
arXiv AI
Oct 27

Topic-Level Fairness in Retrieval-Augmented Generation

The paper investigates topic-level fairness in retrieval‑augmented generation (RAG) systems, noting that aggregate metrics can mask disparities across individual topics. It proposes adapting a reranker per topic using a Bradley‑Terry objective guided by preference pairs derived from the generator’s own output, which reduces mean absolute per‑topic disparity by 51% on RoMQA and 40% on TREC Fair Ranking while preserving accuracy. Existing baselines, even when modified to treat topics separately, fail to achieve comparable reductions.

By Jaeyoung Park, Hantian Zhang, Kexin Rong, Steven Euijong Whang
arXiv AI
Oct 27

Logic Hijacking of RAG: Single-Document Poisoning via Local Optimization

The paper introduces MONOLOGUE, a single‑document poisoning framework that uses logic hijacking to subtly alter Retrieval‑Augmented Generation (RAG) outputs without overt contradictions. Instead of directly refuting facts, MONOLOGUE frames updates as authoritative historical corrections, embedding counterfactuals into a grounded narrative. The method refines payloads offline through local multi‑dimensional gated optimization, combining text quality audits with surrogate behavioral verification, and demonstrates superior attack efficacy across multiple benchmarks, retrieval backends, and open‑weight generators.

By Hangyi Zhao, Bonan Zhang, Fusen Guo, Jiaheng Wei, Chao Chen, Tianqing Zhu
arXiv AI
Oct 27

DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting

arXiv:2610. 34855v1 Announce Type: new Abstract: Visual counting benchmarks report whether vision-language models produce the correct number, but not whether errors arise from perception or reasoning.

By Qiuyi (Richard), Zhang, Zhen Xu, Jing Gu, Seojin Bang, Arnav Mohanty Das, Yaning Hu, Gautam Kumar, Jared Lichtarge, Le Xue, Bettina Catoto, Akul Datta, Daniel Luo, Farha Khan, Marcella Valentine, Yinfei Yang, Andrew M. Dai
arXiv AI
Oct 27

Beyond Behaviour: A Stimulus-Response Safety Evaluation of Vision-Language-Action Models

The paper titled "Beyond Behaviour: A Stimulus-Response Safety Evaluation of Vision-Language-Action Models" examines how vision-language-action models respond to safety-critical stimuli. By testing three models on two video benchmarks, the authors analyze internal representations and action tendencies when faced with visible hazards and harmful outcomes. The results indicate that these models tend to continue task execution even after internally registering danger.

By Vy Vo, Dinh Phung
arXiv AI
Oct 27

Critic's Eye: Review-Grounded Graph Reasoning for Long-Form Motif Discovery

The paper introduces Critic's Eye, a method for computational motif discovery that models a play as a heterogeneous Narrative Topology Graph and performs discriminative inference over global structural evidence. It achieves 84.8% Hit@5 on a benchmark, nearly quadrupling the performance of large foundation models such as Gemini-3-pro-preview. The authors argue that explicit structural priors outperform mere parameter scaling for decoding the complex architecture of long‑form literature.

By Xinyu Sun, Zhangyue Yin, Xinyi Ju, Xin Wang
arXiv AI
Oct 27

Infinity Machine: a self-improving AI agent for research

The paper introduces the Infinity Machine, a self‑improving AI agent designed to solve research problems whose solutions can be programmatically verified. The system operates autonomously: it generates candidate solutions, verifies them, and audits the results without human intervention, all within a sandboxed environment governed by a kernel‑enforced gate. In a six‑day experiment, the Infinity Machine achieved four new state‑of‑the‑art results in quantum information and improved qubit‑routing and hardware efficiency, while also documenting several failure modes.

By Harish Santhanalakshmi Ganesan
arXiv AI
Oct 27

Patient-specific rubric-based evaluation of large language models across longitudinal cancer care

The study introduces NPC‑RubricBench, a phase‑aware benchmark that evaluates large language models (LLMs) on multistage clinical reasoning throughout the nasopharyngeal carcinoma care continuum. Using longitudinal data from 405 patients, the benchmark contains 941 patient‑phase units and 17,688 patient‑specific criteria across five dimensions, focusing strictly on decision‑time evidence. Evaluation of 30 general‑purpose and medical LLMs shows that strong diagnostic accuracy does not translate into better downstream care, with failure modes shifting from explicit decision errors in staging to decision omissions in initial work‑up and follow‑up, and reveals a trade‑off between prompt granularity, decision quality, evidence faithfulness, and safety adherence.

By Wei-Wei Zhang, Pengkai Wang, Yan Li, Zhijie Sang, Guan-Qun Zhou, Jun-Yan Li, Shu-Han Zhou, Pei-Wen Liu, Ting-Xi Tang, Min Tang, Zhitian Hou, Jianmin Wu, Jian Guan, i Lin, Hongxia Yang, Ying Sun