arXiv AI By Tianzhu Zhang

When Should LLMs Trust Their Own Revisions? A Risk-Aware Study of Intrinsic Self-Correction

Read the original on arXiv AI →

The paper investigates intrinsic self‑correction, where a language model revises its own answer without new evidence. Across 29 open‑weight LLMs on BoolQ, GSM8K, and Corr2Cause, the study tracks how revisions change correctness, revealing that while some models improve significantly, others lose a notable fraction of correct answers. The authors compare three runtime strategies—keeping the initial answer, always accepting the revision, and selectively gating revisions—and find that the best approach depends on the model and task, suggesting that self‑correction should be treated as a revision policy rather than a uniformly beneficial second pass.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
3d ago

When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision

The study introduces a two‑pass audit to evaluate how language models revise decisions when faced with disagreement. By fixing the question, evidence, and initial answer, the audit isolates the effect of different revision contexts—neutral reflection, anonymous disagreement, or disagreement from a domain expert—on accuracy. Across five models and 1,989 evidence‑grounded decisions in medicine, science, and law, expert disagreement rarely improves accuracy, often reducing it by up to 16.1 points, and the number of reversals is a poor indicator of harm.

By Yupeng Tang, Mingfeng Lin
arXiv Machine Learning
Sep 25

Return or Revise? Learning When Revision Helps Retrieval-Augmented QA

The paper investigates when it is better to return an existing draft answer or revise it using retrieved evidence in retrieval‑augmented QA systems. By grading both the draft and its candidate revision with the same correctness judge, the authors define a paired effect called recoverability and train policies to predict it before revision. Experiments on 25,870 open‑domain questions show that a recoverability‑based scorer outperforms a draft‑correctness scorer across multiple Llama setups, improving accuracy–revision trade‑offs and closing a significant portion of the oracle gap, though it still applies harmful revisions in a substantial fraction of cases.

By Nicholas Kashani Motlagh, Tim Anderson, Jeremy Gwinnup, Grant Erdmann
arXiv AI
Oct 2

Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systems

The paper introduces Independent–Communicate–Revise (ICR), a framework that isolates communication effects in large language model multi‑agent systems by fixing initial reasoning and measuring how messages influence answer revision. ICR evaluates correction, preservation, and selectivity across four reasoning benchmarks, revealing that similar overall accuracy can mask divergent revision behaviors. The study shows that richer messages can both improve and harm outcomes, and that receiver policies can shift preservation and correction dynamics differently across tasks.

By Shixuan Li, Wei Yang, Peiyu Zhang, Anzhe Cheng, Heng Ping, Paul Bogdan
arXiv Computation and Language
Sep 24

Exact Feedback Is Not Control: Evaluating Text-based Closed-Loop Revision in LLMs

The paper introduces a fixed‑budget revision protocol that uses deterministic verifiers to expose all remaining violations across exact‑length, lexical, and compositional constraints, thereby isolating model‑side revision behavior. Experiments on 19 open‑ and closed‑source LLMs show wide variability in controller‑level success, with some models achieving up to 99.8% success while others remain below 20%. Controlled studies reveal that post‑training and scale affect model responses to exact feedback, but do not consistently improve exact correction, and that recurrence of earlier outputs is linked to lower recoverability.

By Haitong Jiang, Chunlin Liu, Yile Wang, Yuhong Feng
arXiv AI
Oct 21

Correcting Attribution Patching with a Few Exact Measurements

The paper introduces Guided-AP, a method that improves attribution patching by using a few exact measurements to identify and correct its errors. On a single prompt, Guided-AP reduces error by up to 26% across three behaviors, while on many prompts it cuts average error on factual completion by 75% to 79%. The approach leverages shared error signs across components and tasks to efficiently guide verification and correction.

By Luyang Zhang, Minxing Zheng, Jialu Wang