The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper investigates whether giving AI monitors access to the final answer improves their ability to verify reasoning. Using 237 step‑by‑step solutions to physics exam questions, the authors found that answer access mainly helps monitors detect inconsistencies with the final answer rather than independently checking the reasoning. Certification of the answer increased overall accuracy and error localization but reduced the ability to flag critical traces where the answer was correct but the reasoning was flawed.
The study investigates the role of rationales in role‑specialized question‑answering pipelines by fixing evidence and candidate answers while varying only the rationale sent from a reasoner to a verifier. Experiments on 400 examples from MuSiQue, HotpotQA, and 2WikiMultiHopQA show that faithful rationales add almost no answer accuracy, whereas corrupted rationales significantly alter support judgments, especially under explicit rationale‑checking prompts. Human audits reveal that many corrupted rationales lead to mistrust or rejection, highlighting the importance of evaluating rationale sharing as a verification mechanism rather than merely a means to improve accuracy.
arXiv:2503. 13445v3 Announce Type: replace-cross Abstract: When asked to explain their decisions, LLMs can often give explanations which sound plausible to humans.
EviScope is a new paired counterfactual benchmark that evaluates grounded language models by fixing the question while manipulating evidence—adding, removing, distracting, or contradicting it. The v1.1 dataset includes 40 four‑condition quartets with repaired counterfactual claims and span‑level support labels for automated assessment. Experiments on Qwen2.5‑7B, Llama 3.1 8B, and Gemini 3.5 Flash show that paired metrics reveal grounding behaviors hidden by simple answer accuracy, such as unsupported answers, conflict blindness, and incorrect non‑answer actions.
arXiv:2608.29956v1 Announce Type: new Abstract: Large language models often answer complex reasoning questions without revealing intermediate steps, raising whether they reason latently or complete p...
arXiv:2609.21996v1 Announce Type: new Abstract: Large language models can hold knowledge they do not report. A model may sandbag on a capability evaluation, or answer against what it internally knows...