arXiv AI
Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 10
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
arXiv:2607. 08456v1 Announce Type: cross Abstract: A model should refuse two different things: answers it would get wrong, and questions it should not answer at all, such as unanswerable ones or ones resting on a false premise.
arXiv Machine Learning
Aug 4
Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
arXiv:2507. 02778v3 Announce Type: replace-cross Abstract: Although large language models (LLMs) have transformed AI, they still make errors and follow unproductive reasoning paths.
arXiv AI
Jun 19
Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning
arXiv:2606. 19808v1 Announce Type: new Abstract: Test-time reasoning is increasingly used as a serving-time control knob, but extra reasoning is not uniformly valuable: it can repair failed attempts, waste compute on already-correct answers, or introduce harmful answer changes.
arXiv AI
Aug 26
Confidently Wrong, Silently So: Auditing Undetectable Failures of a Deployed On-Device Language Model
The paper audits a developer‑accessible on‑device language model, revealing that it can confidently produce incorrect answers while refusing benign prompts, a phenomenon termed task‑asymmetric miscalibration. The model’s confident outputs are surface‑indistinguishable, with classifiers based on user‑visible features failing to separate correct from wrong responses. The authors propose a model‑agnostic audit protocol, a surface‑indistinguishability test, and a black‑box consistency wrapper that improves reliability without requiring model access.
arXiv AI
Jul 2
Theoria: Rewrite-Acceptability Verification over Informal Reasoning States
arXiv:2607. 01223v1 Announce Type: new Abstract: When should an AI system's answer be trusted?
arXiv AI
Sep 7
When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models
A 0.6B language model consistently answers YES to 1,200 logical tests, yet its behavior shows no discrimination. Linear probes reveal the correct verdict with high AUC (0.96) and transfer to unseen structures, but a single scalar readout fails due to a saturated decision threshold offset by +4.6 σ. Adjusting this threshold restores behavior accuracy from 50 % to 81 % and improves higher‑scale models, demonstrating that miscalibrated readouts, not hidden knowledge loss, drive performance gaps.