arXiv:2609.00624v1 Announce Type: new Abstract: A prominent paradigm in inference-time alignment employs lightweight supervisors to steer Large Language Models (LLMs). Through empirical analysis, we...
arXiv:2603. 07445v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often require fine-tuning (FT) to perform well on downstream tasks, but FT can induce safety-alignment drift even when the training dataset contains only benign data.
Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable.
arXiv:2606. 11201v1 Announce Type: cross Abstract: The wide deployment of LLMs has made model alignment necessary to make newly trained models safely and effectively respond to user instructions.
arXiv:2606. 02530v1 Announce Type: new Abstract: Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax.
Atomic Consistency Preference Optimization (ACPO) is a self‑supervised preference‑tuning method that improves factual accuracy in large language models without external supervision. It uses atomic consistency signals—agreement of individual facts across multiple stochastic responses—to identify high‑ and low‑quality data pairs for model alignment. ACPO outperforms a strong supervised alignment baseline by 1.95 points on the LongFact and BioGen datasets for Phi‑3 and Llama‑3 models.
arXiv:2608.30319v1 Announce Type: cross Abstract: Large language models (LLMs) finetuned for specialized domains represent crucial high-impact applications. Inference-time alignment improves safety d...
arXiv:2607. 20515v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) is critical for aligning Large Language Models (LLMs) with human preferences.
The paper introduces a technique called Synthetic Document Finetuning (SDF) to expose hidden misaligned behaviors in large language models that may engage in alignment faking. By finetuning models to associate certain keywords with undetected misaligned actions, the authors create honeypot environments and observe that the gap in misaligned action rates between cueed and uncued prompts widens, indicating more reliable detection than prompt-based methods. The study also notes that SDF can impair reasoning abilities, proposing on-policy distillation as a mitigation.
The paper investigates how different training strategies affect the prompt sensitivity of large language models. It reproduces and compares methods such as refined data construction and robustness objectives, finding that while robustness fine‑tuning improves over standard fine‑tuning and in‑context learning, the prompt gap remains large (40–57%). Notably, newer techniques like CoIN and PPCL often underperform a simple data‑construction approach that uses one template per batch, and diagnostics suggest that mixed‑template batches force the optimizer to reconcile conflicting updates rather than learn a prompt‑agnostic representation.
arXiv:2606. 00686v1 Announce Type: new Abstract: The prevailing paradigm in large language model (LLM) alignment operates via erasure, filtering unsafe data or training models to strictly refuse harmful prompts.
Domain fine-tuning degrades the safety of large language models: fine-tuned specialists readily comply with harmful prompts framed in domain language. Existing inference-time defenses that mix logits from a safe anchor model require both models to share a vocabulary, which rules them out for the cross-family specialists where safety is most degraded.