Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

The paper investigates whether the text of chain‑of‑thought reasoning steps actually reflects their true importance for a model’s final answer. By defining step importance as the advantage in expected reward when a step is included, the authors use Monte Carlo rollouts to estimate ground truth and then test whether large language model judges can identify high‑advantage steps. They find that capable LLMs can beat a prevalence baseline but still fall far short of a noise ceiling, and that fine‑tuning a step‑level critic improves detection for incorrect responses but remains distant from the ceiling for correct ones, indicating that step importance is only partially recoverable from the reasoning trace text.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• llms
• reinforcement-learning
• fine-tuning
• safety