arXiv AI
Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Jun 25
Reasoning Quality Emerges Early: Data Curation for Reasoning Models
Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs). However, existing methods for curating high-quality SFT data rely heavily on strong reasoning models to filter examples based on diversity and difficulty, making the curation process costly while often yielding suboptimal data quality.
arXiv Machine Learning
Jun 26
Reasoning Quality Emerges Early: Data Curation for Reasoning Models
arXiv:2606. 26797v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs).
arXiv Machine Learning
Jul 14
Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts
arXiv:2607. 09999v1 Announce Type: cross Abstract: We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved.
arXiv Machine Learning
Aug 31
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
The paper introduces CodeRQ-Bench, the first benchmark for assessing large language model reasoning quality across coding tasks such as generation, summarization, and classification. It analyzes over a thousand mismatches from existing evaluators, identifies recurring limitations, and derives design insights that lead to a new two‑stage evaluator, VERA. Experiments show VERA outperforms strong baselines, improving AUCROC by up to 0.26 and AUPRC by up to 0.21 on four datasets.
Hugging Face Trending Papers
Sep 24
Reasoning Instructions Can Break Answer Decoding in Vision--Language Models
The paper shows that chain‑of‑thought (CoT) instructions can distort multiple‑choice vision‑language model evaluation when a scorer appends a reasoning cue but reads answer‑label logits before the model generates any rationale. This CoT‑prefix scoring causes significant drops in accuracy (e.g., Qwen2.5‑VL‑7B falls from 80.76% to 45.48% on ScienceQA) and leads most predictions to choose the first option. Analysis reveals that while answer information remains linearly accessible in late layers, the immediate readout is misled by probability mass shifting toward continuation tokens, and the issue varies across datasets and models.
arXiv AI
Sep 25
Reasoning Instructions Can Break Answer Decoding in Vision--Language Models
The paper shows that chain‑of‑thought (CoT) instructions can distort evaluation of vision‑language models (VLMs) when a scorer reads answer‑label logits before the model generates a rationale. On ScienceQA, Qwen2.5‑VL‑7B’s accuracy falls from 80.76% to 45.48% under this CoT‑prefix scoring, and most predictions incorrectly pick the first option. Linear probes and free generation recover most of the lost accuracy, indicating that the answer information remains in the hidden states but is missed by the early readout. The authors explain the mismatch with vocabulary and layer diagnostics, noting that probability mass shifts toward continuation tokens while answer information stays linearly accessible in later layers. The effect varies across datasets and models, but the study demonstrates that CoT‑prefix scoring can misrepresent model knowledge unless the requested and scored outputs are aligned.