arXiv AI
Jun 3
Consistency Training Can Entrench Misalignment
arXiv:2606. 03810v1 Announce Type: cross Abstract: Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures.
Consistency Training Can Entrench Misalignment
Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures. Such methods are simple, scalable, and largely label-free, but their effects on model alignment remain poorly understood.
Related stories
arXiv Machine Learning
Jul 30
Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models
arXiv:2607. 26173v1 Announce Type: new Abstract: Alignment training, model organisms, and toy models are usually treated as separate research areas.
arXiv Machine Learning
Aug 6
Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning
arXiv:2608. 04347v1 Announce Type: new Abstract: Fine-tuning enables a source model to acquire desired capabilities and behaviors in a target domain while retaining much of its general-purpose competence.
arXiv AI
Jul 3
Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
arXiv:2510. 17426v3 Announce Type: replace-cross Abstract: The "alignment tax" of post-training is typically framed as a drop in task accuracy.
Hugging Face Trending Papers
Aug 5
Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning
Fine-tuning enables a source model to acquire desired capabilities and behaviors in a target domain while retaining much of its general-purpose competence. However, this adaptation process can also degrade alignment properties that were present in the source model.
arXiv AI
Jun 11
To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending
arXiv:2606. 11201v1 Announce Type: cross Abstract: The wide deployment of LLMs has made model alignment necessary to make newly trained models safely and effectively respond to user instructions.
Hugging Face Trending Papers
Jun 17
The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs
Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment.
OpenAI Blog
Jun 18, 2025
Toward understanding and preventing misalignment generalization
We study how training on incorrect responses can cause broader misalignment in language models and identify an internal feature driving this behavior—one that can be reversed with minimal fine-tuning.
arXiv Machine Learning
Jul 13
Hair-Trigger Alignment: Black-Box Evaluation Cannot Guarantee Post-Update Alignment
arXiv:2601. 22313v2 Announce Type: replace Abstract: Large Language Models (LLMs) are rarely static and are frequently updated in practice.
arXiv AI
Jul 1
The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes
arXiv:2606. 30653v1 Announce Type: cross Abstract: Large language models are increasingly deployed in agentic pipelines that depend on the model evaluating its own outputs without external verification.
arXiv Machine Learning
Jun 5
In-Training Defenses against Emergent Misalignment in Language Models
arXiv:2508. 06249v3 Announce Type: replace Abstract: Fine-tuning lets practitioners repurpose aligned large language models (LLMs) for new domains, yet recent work reveals emergent misalignment (EM): Even a small, domain-specific fine-tune can induce harmful behaviors far outside the target domain.
arXiv AI
Jun 19
Emergent Alignment
arXiv:2606. 19527v1 Announce Type: new Abstract: Can Large Language Models (LLMs) discern when their own outputs are misaligned with human ethics?