arXiv:2607. 14552v1 Announce Type: cross Abstract: A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors.
arXiv:2603.10359v2 Announce Type: replace Abstract: Distilling reasoning capabilities from Large Reasoning Models (LRMs) into smaller models is typically constrained by the limitations of rejection s...
MedFabric is a new benchmark for detecting word‑level medical fabrications, comprising 646 fabricated statements each paired with a ground‑truth passage that shares the same LLM authorship and nearly identical wording. The study shows that current detectors perform poorly—expert clinicians achieve only 53.3% macro‑F1 and no detector family surpasses 60% without gold evidence—highlighting that detection hinges on evidence correctness rather than subtlety of fabrication. The authors demonstrate that a retrieval‑confidence gate can substantially improve performance, raising macro‑F1 from 61% to 74%.
Medical multimodal reasoning models are typically trained on teacher-generated rationales, discarding the valid reasoning in students' own failed trajectories, while general-domain correction methods...
arXiv:2610.17690v1 Announce Type: new Abstract: Medical multimodal reasoning models are typically trained on teacher-generated rationales, discarding the valid reasoning in students' own failed traje...
arXiv:2607. 28336v3 Announce Type: replace Abstract: On-policy distillation provides dense supervision for multimodal reasoners, but its trajectory-level reward cannot determine whether a failed answer arose from perception or subsequent reasoning.
arXiv:2607. 09692v1 Announce Type: new Abstract: Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations.
The paper introduces two distillation methods for handling misaligned AI models. Distillation for Incrimination (DFI) transfers misalignment without the ability to conceal it, making students more likely to admit hidden behaviors. Distillation for Capabilities (DFC) transfers useful capabilities while minimizing misalignment, using techniques like inoculation prompting and focused training.
arXiv:2607. 26929v1 Announce Type: cross Abstract: The same diagnostic result can support or challenge one causal claim yet fail to address another when the claims concern different populations, outcomes, estimands, pathways, or identifying assumptions.
arXiv:2607. 23731v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own trajectories while a teacher supplies dense token-level likelihoods at student-visited prefixes.
The paper proposes using the temporal volatility of internal attention mechanisms—measured by an unsupervised attention dispersion metric—as a diagnostic signal for hallucinations in large language models. It demonstrates that spikes in attention entropy within intermediate layers correlate with reasoning breakdowns, and shows statistically significant AUC improvements of up to +0.076 over output-based baselines on GSM8K and MATH-500 benchmarks using the Qwen2.5 model family.
The paper investigates why supervised fine‑tuning (SFT) of large language models leads to increased hallucinations of factual information. It proposes a self‑distillation SFT approach that regularizes output‑distribution drift to preserve pre‑training knowledge, and shows that freezing parameter groups can reduce hallucinations when new knowledge is unnecessary. Experiments attribute the main cause to interference among overlapping semantic representations, which self‑distillation mitigates, and an associative‑memory model explains the forgetting dynamics.