Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought

The study evaluates whether the chain-of-thought (CoT) rationales produced by medical language models truly influence their answers. Using a 30‑operator perturbation audit that modifies both the question and the CoT (e.g., severity reversal, negation flip, demographic swap, evidence ablation), the authors found that 72.9% of edits did not change the model’s answer—a high Chain‑Decoupling Rate (CDR). Across 14 large language models and four medical QA benchmarks, the CoT text had little impact on accuracy, and removing CoT prompting did not reduce performance. "whyItMatters":"The findings suggest that current medical CoT outputs may be more documentation than genuine reasoning, highlighting the need for better faithfulness checks in clinical AI systems."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• fine-tuning
• benchmarks