AI safety and alignment

Alignment, interpretability, red-teaming, bias and privacy: the research on what these systems do when they misbehave.

17,788 stories · RSS feed

arXiv AI
3d ago

Reject, Remember, Retry: Why Multi-Turn Attackers Should Remember Their Mistakes

The paper introduces $R^3$ (Reject, Remember, Retry), a black-box multi-LLM framework that improves multi-turn jailbreak attacks by conditioning on a compact state instead of the full conversation transcript. In $R^3$, failed attack turns are removed from the target’s context but stored by the attacker along with the reason for failure, creating an asymmetric memory. Experiments on two benchmarks and six target LLMs show that $R^3$ outperforms state‑of‑the‑art baselines by up to 53% in ASR@5 under matched query budgets, and even a low‑capacity attacker using $R^3$’s state can surpass high‑capacity attackers without it by up to 34% in ASR@5.

By Jekaterina Novikova, Yuqing Huang, Jack Ji, Sagarika Suresh, Jithin Pradeep, Lovedeep Gondara
arXiv AI
3d ago

Specification-Aware Distribution Shaping for Robotics Foundation Models

The paper introduces a specification‑aware action distribution optimization framework that enforces Signal Temporal Logic (STL) constraints during the execution of pretrained robotics foundation models without altering their parameters. At each decision step, the method minimally adjusts the action distribution to satisfy hard STL feasibility constraints by reasoning over the remaining horizon with forward dynamics propagation. The approach is validated in simulation using state‑of‑the‑art robotics foundation models across multiple environments and complex specifications.

By Sad{\i}k Bera Y\"uksel, Derya Aksaray
arXiv AI
3d ago

ThinkSafe: Self-Generated Safety Alignment for Reasoning Models

ThinkSafe is a method for aligning large reasoning models with safety objectives without relying on external teachers. It uses a frozen student model to generate safety-alignment data through refusal steering and a safety guard, then fine‑tunes offline. Experiments on DeepSeek‑R1‑Distill and Qwen3 demonstrate significant safety gains while maintaining competitive reasoning performance, and on Qwen3‑0.6B it outperforms GRPO in roughly one‑seventh the wall‑clock time.

By Seanie Lee, Sangwoo Park, Yumin Choi, Gyeongman Kim, Minki Kang, Jihun Yun, Sung Ju Hwang, Dongmin Park, Jongho Park
arXiv AI
3d ago

When Safety Checks Become Privacy Risks: A Privacy-Preserving Paradigm for LLM Agent Safety Auditing

The paper introduces a privacy‑preserving safety auditing paradigm for large language model agents that interact with online services via tool calls. It separates audit computation from plaintext access, using trusted representation construction, encrypted audit computation, and trusted decision interfaces to protect sensitive user data. The authors implement this paradigm in SealCall, which employs learned audit representations and CKKS‑based encrypted similarity computation, achieving a mean F1 score of 0.840 on 1,252 public binary candidates while maintaining plaintext audit decisions and completing encrypted scoring in 0.176 s.

By Yixiang Luo, Ziheng Ma, Ke Chang, Jian Wang, Jiqiang Liu
arXiv AI
3d ago

Instruction-Tuning Open-Weight Language Models for BPMN Model Generation

The paper presents InstruBPM, an instruction‑tuned open‑weight large language model that generates BPMN process models from natural language descriptions. By preparing paired text‑diagram data and applying parameter‑efficient fine‑tuning with quantization, the authors demonstrate that the tuned model outperforms both untuned open‑weight baselines and strong proprietary models in sequence, structural, and guideline‑conformance metrics. The approach is cost‑effective, privacy‑preserving, and supports on‑prem deployment, with publicly released models and scripts for reproducibility.

By G\"okberk \c{C}elikmasat, Atay \"Ozg\"ovde, Fatma Ba\c{s}ak Aydemir
arXiv AI
3d ago

SecPO: Principled Adversarial Training for Prompt Injection Security

SecPO introduces a principled adversarial training framework to defend large language models against prompt injection attacks. By approximating an ideal injection‑invariant reference model and employing a new SecPO loss, the authors improve security while preserving utility. Combined with an efficient adversarial training pipeline that synthesizes strong injected inputs, the resulting SecPO + AT model significantly reduces attack success rates compared to existing defenses and maintains performance across multiple benchmarks.

By Sizhe Chen, Yibo Peng, Jaewon Chang, Chawin Sitawarin, David Wagner
Hugging Face Trending Papers
4d ago

D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models

The paper introduces D&A (Describe & Assess), a prompt‑based method that separates vision‑language model progress estimation into a textual description of scene dynamics and a rubric‑based assessment. By requiring the model to explicitly define absolute success before scoring, D&A reduces confirmation bias and provides fine‑grained, partial progress metrics that accurately penalize failures and interpolate successes. The approach generalizes across tasks, embodiments, and benchmarks, matching or surpassing post‑trained reward models while enabling efficient downstream reinforcement learning.

Hugging Face Trending Papers
4d ago

MultiInpaintCOCO: Fine-Grained Visual Understanding in Multiple Languages

MultiInpaintCOCO extends the English InpaintCOCO dataset to 13 typologically diverse languages, creating a multilingual benchmark of 953 minimal-pair examples focused on fine-grained visual details. The study evaluates state-of-the-art multilingual multimodal retrieval models, revealing that performance drops significantly for lower-resource languages and that many models exhibit a systematic bias toward English captions. The benchmark and code are publicly available at https://github.com/ufal/multi-inpaint-coco.

Hugging Face Trending Papers
4d ago

Specificity Bias and the Wikipedia Paradox: An xAI Analysis of AI Text Detectors under Domain Shift

The paper examines how fine‑tuned Transformer detectors (DistilBERT, RoBERTa) perform on an out‑of‑distribution dataset of hyper‑local news generated by LLMs. Using SHAP‑based explainable AI, it identifies three phenomena: a reliance on a “Call‑to‑Action” syndrome, a “Specificity Bias” where hallucinated named entities evade detection, and a “Wikipedia Paradox” where formal human text is misclassified as AI. These results highlight systematic shortcomings of purely statistical detection methods in structured human domains.

Hugging Face Trending Papers
4d ago

Unlocking the Potential of Vision Foundation Models for Multi-Task Dense Prediction via Mixture of Task Adapters

The paper introduces D2MoE, a Mixture of Task Adapters designed for vision foundation models to handle multi-task dense prediction. It separates adaptation into a task‑generic low‑rank channel (TG‑LoRA) that preserves pretrained priors and a task‑adaptive sparse expert module (TA‑MoE) that resolves task‑specific conflicts. A Semantic Prior Injector further aligns multi‑level task features with frozen foundation knowledge, achieving robust multi‑task performance with a small trainable footprint.

Hugging Face Trending Papers
4d ago

EffComp: Efficient Prompt Compression via Hybrid Reinforcement & Supervised Learning

EffComp is a framework for efficient prompt compression in RAG-based question answering. It uses BERT-style document reranker and sentence selector models to perform fast, extractive compression at the sentence level, thereby avoiding hallucinations. The hybrid reinforcement and supervised learning training optimizes compression ratio while maintaining LLM accuracy, and experiments show it outperforms existing methods with minimal latency.

Hugging Face Trending Papers
4d ago

3L-D: A Low-Light Level Dataset from a Sensor-Agnostic, Alignment-Preserving Acquisition Framework

The paper introduces 3L-Dataset, a physically calibrated low‑light image and video dataset created with a modular, sensor‑agnostic acquisition framework. The test bench reproduces the Midavaine night‑level scale from full moon to overcast moonless night, covering visible to near‑infrared spectra across varied environments while ensuring sub‑pixel alignment across synchronized image sextuplets. The authors evaluate deep‑learning baselines on this dataset, highlighting trade‑offs between reconstruction fidelity, perceptual quality, and inference latency under severe photon starvation.

Hugging Face Trending Papers
4d ago

PriCBMs: Multi-granular Concept Bottleneck Vision-Language Models for Visual Privacy Recognition

PriCBMs introduces Privacy Concept Bottleneck Models that use intermediate privacy-related concepts to identify category-relevant privacy cues via visual-concept matching before predicting privacy categories. The approach incorporates a concept selection strategy based on relevance, discriminability, and diversity, and adds a global-region modeling module to capture localized privacy evidence while preserving global context. Experiments on VISPR and FineViP demonstrate that PriCBMs outperform competitors across all four multi-label metrics, with ablation studies confirming the effectiveness of the design.

Hugging Face Trending Papers
4d ago

Latent Diffusion-GAN: Adversarial Learning in the Autoencoded Latent Space

Latent Diffusion-GAN (LD‑GAN) integrates adversarial learning into diffusion models by using the pretrained VAE as an energy‑based discriminator, avoiding extra parameters and preserving the latent priors. It introduces a structural consistency energy to align encoder and decoder features, improving perceptual quality. Experiments show LD‑GAN enhances sample fidelity, sharpness, and diversity across generation tasks while maintaining efficient training dynamics.

arXiv AI
4d ago

LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling

The paper introduces Transition-Aware Reward-Guided Representation Sampling (TARGRS), a test-time detoxification framework for large language models that learns reward guidance over toxicity transitions in latent representation space. TARGRS constructs trajectories from non‑toxic to toxic states, uses toxicity scores along these paths to train a token‑level reward model, and then guides coarse‑to‑fine sampling in the opposite direction to produce detoxified outputs with minimal perturbation. Experiments on ten LLMs and three LVLMs show that TARGRS consistently outperforms existing baselines, reducing toxicity by up to 69.38% on RealToxicityPrompts and up to 64.23% on LVLMs while maintaining near‑original speed and core capabilities.

By Aishan Liu, Yisong Xiao, Xianglong Liu, Yongxin Huang, Siyuan Liang, Zonghao Ying, Yang Qu, Jian Yang, Yaodong Yang, Dacheng Tao
arXiv Computation and Language
4d ago

Table Serialization Determines Which Relations Language Models Preserve

The paper investigates how the linear serialization of tables influences the ability of instruction‑tuned language models to perform table lookup tasks. By comparing row‑major and column‑major formats on synthetic 256‑cell tables of varying aspect ratios, the authors find that the optimal serialization depends on table shape: row‑major works best for tall tables, while column‑major excels on wide tables. Error analysis reveals a complementary pattern—row‑major preserves the correct row but often misses the target column, whereas column‑major preserves the correct column but sometimes loses the row—leading the authors to introduce the concept of relational locality. "whyItMatters":"The study shows that failures attributed to table width are actually driven by the chosen serialization format, highlighting the importance of representation choices for language model performance on tabular data."

By Junwei Ma, Arun Kumar Chithanar, Palash Goyal, Chul Lee
arXiv Machine Learning
4d ago

New Lower Bounds for Private PAC Learning

The paper investigates realizable PAC learning under approximate differential privacy and develops a framework for establishing lower bounds on sample complexity. It demonstrates that classes sharing the same VC and Littlestone dimensions can have different private sample complexities, introducing the Independent Class Packing (ICP) parameter to capture this distinction. Using ICP, the authors derive tight lower bounds for several hypothesis classes, including axis-aligned rectangles, unions of intervals, linear subspaces, and improved bounds for halfspaces and convex polygons, and show that ICP characterizes private PAC sample complexity for product classes up to logarithmic factors.

By Prathamesh Dharangutte, Amartya Sanyal
arXiv Machine Learning
4d ago

CausalXRL: Explainable Reinforcement Learning through Causal Graph Reasoning

CausalXRL is a new framework that makes reinforcement learning agents explainable by using causal graph reasoning. It offers transparent, structured, multi‑level explanations of agent decisions. The authors validate the approach with quantitative tests on benchmark RL environments and a qualitative expert study on the real‑time strategy game MicroRTS, showing faithful explanations and efficient causal‑structure learning.

By Yanming Zhang, Eric Papenhausen, Klaus Mueller