YFPO (Yoked Feature Preference Optimization) is a neuron‑guided preference optimization framework that augments standard preference learning with internal neuron‑level rewards. It uses AttnLRP to identify math‑associated internal features and derives an auxiliary reward from the activation margin between preferred and dispreferred responses. Experiments on GSM8K with a compact language model show that these neuron‑guided rewards influence optimization dynamics and yield measurable improvements, indicating that internal representations can serve as lightweight, interpretable signals for reasoning‑oriented post‑training.
arXiv:2607. 02460v1 Announce Type: cross Abstract: Post-training large language models (LLMs) without real-world interaction feedback or human-labeled supervision remains challenging, particularly in specialized domains where expert annotations are costly to obtain.
Achieving strong optimization generalization across diverse optimization problems while requiring limited training resources remains a challenging problem for optimization-oriented large language models (LLMs). Existing approaches typically rely on large-scale supervised datasets, costly reasoning annotations, and expensive intermediate step verification, resulting in substantial training overhead.
The paper introduces INSPIRE, an Internalize-Then-Improve framework designed to enhance example-driven mathematical reasoning in large language models. It combines Reference-Guided Student Internalization (RGSI) to generate high-quality preference pairs with a staged rubric preference training that separates method learning from correctness. Experiments across various model sizes show consistent gains, even outperforming larger open-source models, and maintain performance on out-of-distribution mathematical tasks.
Post-training large language models (LLMs) without real-world interaction feedback or human-labeled supervision remains challenging, particularly in specialized domains where expert annotations are costly to obtain. Recent annotation-free self-evolution methods address this by using the model's own outputs as supervision signals, constructing a teacher via additional context and aggregating predictions across multiple rollouts through majority voting to produce pseudo-labels.
arXiv:2607. 11505v2 Announce Type: replace-cross Abstract: Post-training for large language models typically couples policy exploration with model optimization, hindering the reuse of high-reward behaviors from policy exploration.
arXiv:2606. 12360v1 Announce Type: new Abstract: Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata.
arXiv:2609.21432v1 Announce Type: new Abstract: Post-training plays a pivotal role in enhancing the reasoning capabilities and task-specific expertise of large language models (LLMs). Despite recent...
arXiv:2605.29089v2 Announce Type: replace-cross Abstract: Recent reinforcement learning (RL) post-training approaches primarily optimize the final output policy using sparse outcome-level rewards, wh...
arXiv:2506. 13702v4 Announce Type: replace-cross Abstract: Single-trajectory preference optimization methods learn from datasets of ((prompt, response, reward)) tuples, offering a practical alternative to pairwise preference learning by directly leveraging scalar feedback.
arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.
arXiv:2503. 00539v2 Announce Type: replace-cross Abstract: Reinforcement learning from human feedback (RLHF) has evolved to be one of the main methods for fine-tuning large language models (LLMs).
arXiv:2603. 19294v4 Announce Type: replace Abstract: While post-training has successfully improved large language models (LLMs) across a variety of domains, these gains heavily rely on human-labeled data or external verifiers.