arXiv AI September 10, 2026 By James Wedgwood, Aashiq Muhamed, Mona T. Diab, Virginia Smith

DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment

The paper introduces DSPA, a dynamic sparse autoencoder (SAE) steering technique that aligns language model outputs with user preferences during inference, avoiding costly weight updates. DSPA constructs a conditional-difference map from preference triples to adjust token-active latents, improving MT‑Bench scores and matching AlpacaEval performance on models like Gemma‑2 and Qwen3 while preserving accuracy. It demonstrates robustness with limited preference data, outperforms the two‑stage RAHF‑SCIT pipeline in FLOPs, and reveals that preference directions are largely driven by discourse and stylistic cues.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 11

Toward Preference-aligned Large Language Models via Residual-based Model Steering

arXiv:2509. 23982v2 Announce Type: replace-cross Abstract: Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences.

By Lucio La Cava, Andrea Tagarelli

llms reinforcement-learning benchmarks safety

arXiv AI

1d ago

A Zeroth-Order Paradigm for LLM Preference Alignment

The paper introduces Comparison-based Preference Optimization (ComPO), a zeroth-order method that aligns large language models with human preferences using comparison oracles instead of direct differentiable loss optimization. It provides theoretical convergence guarantees for both offline and online variants under smoothness, gradient sparsity, and oracle compatibility assumptions, and establishes performance bounds under local coverage and in-distribution reward accuracy. Experiments on several LLMs (Mistral, Llama, Gemma-2, Qwen3, Gemma-3) show that ComPO outperforms existing direct alignment methods, achieving higher length-controlled win rates and diagnostics that suggest mitigation of likelihood displacement.

By Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin

llms fine-tuning efficiency safety

arXiv AI

Jun 2

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.

By Hyung Gyu Rho

llms nlp reinforcement-learning

arXiv AI

Aug 17

EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignment

arXiv:2608. 14011v1 Announce Type: cross Abstract: Generative recommendation autoregressively generates the semantic IDs of the target item, unifying preference modeling and index retrieval within the shared token space.

By Haokai Ma, Aoqi Hu, Yueao Xing, Ruobing Xie, Yonghui Yang, Teng Tu, Lei Meng, Tat-Seng Chua

safety

arXiv Machine Learning

Jun 10

Mechanistic Analysis of Alignment Algorithms in Language Models

arXiv:2606. 09850v1 Announce Type: new Abstract: Post-training alignment algorithms are predominantly evaluated as black boxes, obscuring how they reshape language models' internal computations.

By Aarush Sinha, Ishan Garg, Veeraraju Elluru, Arth Singh, Kushal Garg

llms diffusion reinforcement-learning safety

arXiv AI

Jun 11

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

arXiv:2605. 12288v3 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions.

By Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa Doan, Trung Le

llms nlp reinforcement-learning benchmarks safety

One email a morning, machine-written

Related stories

Toward Preference-aligned Large Language Models via Residual-based Model Steering

A Zeroth-Order Paradigm for LLM Preference Alignment

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization

EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignment

Mechanistic Analysis of Alignment Algorithms in Language Models

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching