The paper introduces LLM-as-an-Investigator, an evidence-first approach for interactive problem diagnosis that mitigates premature commitment to incorrect hypotheses. It implements this through the Solution Investigator Agent (SIA), which coordinates a Diagnostic Reasoner, a Hypothesis Investigator, and a Counterfactual Investigator to gather discriminative evidence and refine competing explanations. The method is evaluated on a structured corpus of 303 technical forum threads, outperforming several baseline strategies and demonstrating robustness to misleading user hypotheses.
arXiv:2606. 13220v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as interactive assistants for technical problem solving.
Large language models (LLMs) are increasingly used as interactive assistants for technical problem solving. However, when users provide incomplete descriptions or plausible but unverified explanations, LLMs may prematurely align with these assumptions and propose solutions before collecting sufficient evidence.
arXiv:2608. 12877v1 Announce Type: new Abstract: Multi-hop fact verification, which verifies claims by reasoning over multiple pieces of evidence, is critical for combating misinformation on social media yet remains highly challenging.
arXiv:2610.14381v1 Announce Type: new Abstract: Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system...
PsyCIDRA is a dual‑agent framework that couples a free‑form psychiatric interviewer with a diagnostic reasoning agent to support expert review. The interviewer agent uses tools to keep working notes, load expert skills, and pull ICD‑11 references, while the diagnostic agent receives the interview transcript and generates hypotheses with supporting, conflicting, and missing evidence, withholding a final hypothesis if insufficient support exists. In simulations and a blinded human study, PsyCIDRA achieved higher diagnostic agreement and rank‑1 accuracy than direct prompting, indicating its promise for assisting psychiatric assessment through interactive dialogue.
BASIS is a benchmark that tests large language model agents on their ability to infer users’ mental states, detect belief‑reality mismatches, and use this insight to craft better solutions in interactive settings. The study shows that current models often ignore users’ mental states, leading to poorer outcomes, while explicitly eliciting and incorporating such information improves solution quality. Additional experiments compare BASIS with Chain‑of‑Thought prompting, targeted training, and multi‑agent collaboration to demonstrate the practical benefits of mental‑state awareness.
arXiv:2606. 10296v1 Announce Type: cross Abstract: Multi-agent debate systems are typically evaluated only on whether the final answer is correct, overlooking the quality of the intermediate reasoning that debate is designed to produce.
arXiv:2607. 17188v2 Announce Type: replace Abstract: While test-time scaling improves the problem-solving ability of large reasoning models (LRMs) through additional inference-time computation, it can also exacerbate overthinking and underthinking, which we formulate as reasoning state--action mismatch.
arXiv:2607. 03847v1 Announce Type: new Abstract: Understanding why discovered scenarios become critical in scenario-based testing is essential for effectively leveraging them in decision-making systems.
arXiv:2605. 21347v3 Announce Type: replace Abstract: Diagnosing failures in LLM agents remains largely manual.
arXiv:2605.11388v2 Announce Type: replace Abstract: Humans intuitively solve complex problems by flexibly shifting among reasoning modes: they plan, execute, revise intermediate goals, resolve ambigu...
The paper introduces a framework to evaluate large language models (LLMs) on their ability to conduct full scientific inquiry cycles, combining hypothesis generation, reasoning, and exploration. By extending the Countdown task into a mathematical environment, the authors assess how LLMs probe systems, collect observations, and infer relationships, examining exploration strategies and causal discovery. Experiments across seven LLMs and various prompting methods show limited inherent inquiry skills, but structured guidance inspired by educational scaffolding markedly improves performance.