arXiv AI
4d ago

Diagnosing and Improving Probabilistic Reasoning in Large Language Models

The paper introduces a decision‑theoretic framework that splits a large language model’s decision loss into belief formation and action selection components. Using a synthetic benchmark, it evaluates how reinforcement‑learning interventions on beliefs, decisions, or both affect these components across three domains. The study finds that targeting a single component improves that part but may not transfer to others, while jointly targeting both improves both only when training and evaluation formats match.

By Huaman Sun, Dingcheng Wang, Jason Hartline, Jessica Hullman
arXiv AI
Sep 10

Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

The paper introduces Fact-Ablated Evaluation (FAE), a framework that iteratively removes cited evidence to test whether large language models (LLMs) adjust their fact‑checking predictions accordingly. Experiments reveal that many off‑the‑shelf LLMs rely more on internal knowledge than on the provided evidence. To address this, the authors propose REAL, a training method that uses counterfactual evidence supervision to encourage LLMs to base veracity judgments on evidence, achieving better evidence‑dependent performance across four datasets.

By Xingyu Deng, Mingzi Cao, Nikolaos Aletras, Xi Wang, Mark Stevenson