MechSparse: Mechanism-Guided Sparse PEFT Selection Is Task-Shaped
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The Flow has not summarised this story yet — read it at arXiv Computation and Language.
MECHVAR is a lightweight, auditable rule for selecting experiments from a finite library to discriminate between candidate mechanisms. It chooses probes by maximizing the posterior‑weighted variance of predicted responses, a score that aligns with the Box–Hill pairwise‑KL criterion and links to expected information gain when separations are small. Experiments on a 25‑block audit and a Digits loop show MECHVAR outperforming confirmation‑first strategies and matching or exceeding EIG in identification accuracy while being far faster to compute.
The paper investigates why large vision‑language models sometimes misclassify harmful memes, attributing failures to either missing internal evidence or poor routing of evidence to the output. Using sparse autoencoders, role‑conditioned probes, and causal interventions on Gemma‑3 and Qwen3.5, the authors show that sparse readouts consistently outperform native predictions across six harmful content benchmarks, revealing a readout gap that is largely due to routing rather than representation. The study also demonstrates that calibration‑only routing recovers most of the performance gap and that the issue persists across languages and is not solely driven by OCR signals.
arXiv:2607. 14111v1 Announce Type: cross Abstract: Can small language models detect and report on perturbations their own internal activations?
arXiv:2607. 21692v2 Announce Type: replace Abstract: Sparse attention prunes a long context to the blocks a model needs, and the usual selector is distilled from a dense teacher's attention.
arXiv:2609.00756v1 Announce Type: new Abstract: The Mutual Reinforcement Effect (MRE) asks whether a fine, span-level and a coarse, document-level task help each other when one model handles both. We...
The study empirically investigates how length‑normalized Direct Preference Optimization (DPO) with a Structured Fine‑Tuning (SFT) retention term (RPO objective) affects downstream capabilities of a Qwen3.5‑4B model fine‑tuned with sparse LoRA on structured financial‑filing tasks. Results show that the alignment tax is graded and task‑selective: extraction performance is largely preserved, JSON extraction degrades modestly, and router‑based tasks suffer severe loss, with most degradation attributed to suppression of rejected completions. The analysis uses 8 seeds, artifact‑aware scoring, and TOST equivalence testing to quantify these effects across different reward margins and precision settings.