How Output Format Confounds Data Quality and Capability in Instruction Tuning
Read the original on arXiv Computation and Language →The paper investigates how the output format of instruction‑tuned models confounds assessments of data quality and model capability. By analyzing gradient signatures across multiple tasks, interfaces, and model families, the authors demonstrate that the interface can mask or exaggerate performance signals, with certain skills appearing only under specific output formats. They show that the interface‑varying residual is not noise but captures task‑specific information, and that correcting a single generation budget can flip the measured effect of fine‑tuning on a benchmark. "whyItMatters":"The study highlights that current evaluation practices may misrepresent a model’s true capabilities because they rely on the output interface rather than the underlying content, potentially leading to misleading conclusions about data quality and model performance."
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.