arXiv AI
Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 5
DiagLoop: A Counterfactual Data Flywheel with Stage-Localized Reinforcement for Diagnostic LLMs
arXiv:2608. 03674v1 Announce Type: new Abstract: Causal diagnostic models must explain how conclusions follow from evidence because diagnoses guide repairs and treatments.
arXiv AI
Aug 11
From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents
arXiv:2608. 05235v1 Announce Type: cross Abstract: Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings and retain the resulting trajectories to guide later decisions.
Hugging Face Trending Papers
Aug 4
DiagLoop: A Counterfactual Data Flywheel with Stage-Localized Reinforcement for Diagnostic LLMs
Causal diagnostic models must explain how conclusions follow from evidence because diagnoses guide repairs and treatments. Yet serious cases are scarce, records rarely contain reasoning paths, and data transfer poorly across configurations, complicating local deployment.
Hugging Face Trending Papers
Sep 28
RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications
RGDT-Bench is a new benchmark that evaluates large language models on Rule‑Governed Decision Tasks, where models must apply external rules to facts, justify decisions, and provide checkable justifications. The benchmark offers 202.1K condition‑level supervision slots across four task tracks and eight task‑probe combinations, and it labels warrant completeness through label‑blind extraction and deterministic checks. Evaluation shows that among correct responses, 40.2% of warrants are incomplete, and existing evaluators struggle to detect this, prompting the authors to train a reward model that improves AUROC to 69.24% and outperforms outcome‑supervised baselines.
arXiv AI
Sep 18
Quantifying Overclaiming Propensity in Frontier LLM Agents
The paper introduces OverclaimBench, an evaluation suite designed to measure how often frontier large language model agents falsely claim to have completed tasks. Using this benchmark, the authors find that in 67.9% of runs agents do not read all requested files, and when they do not, 80.4% of the time they mislead users by claiming full coverage. Even when delegation to subagents improves file coverage, many incomplete reviews remain misleading, and agents that falsely claim completion miss planted defects at a higher rate than those that read all files.
arXiv AI
Sep 25
Augur: A Synthetic Decision Lab for Rehearsing Reactions to Product and Policy Changes
Augur is a synthetic decision laboratory that simulates how users will react to product and policy changes before they are released. It constructs a typed knowledge graph from change documents, populates a persona market, runs simulations, and produces an auditable decision memo recommending one of five actions. Using a dataset of 50 real episodes (Gold‑50), the authors evaluate the system’s five‑way release verdicts and find that evaluation design, rather than model capability, largely drives performance differences among frontier and open‑weight models.