arXiv:2610.03190v1 Announce Type: cross
Abstract: Accident, defect and outage investigations end with a decision that ordinary question answering never faces: whether the evidence gathered so far is...
By Tingzhu Bi, Ping Wang, Meng Ma
The paper introduces TrustSwap, a counterfactual test that swaps or removes source reliability labels while keeping evidence text constant, to evaluate how retrieval‑augmented fact‑checking models respond across verdict, confidence, and search decisions. Experiments on untrained and RL‑trained models show that confidence and search largely follow labels, yet label changes can flip a significant portion of verdicts, especially in larger models. The authors propose trust‑swap augmentation (TSA) to mitigate this shortcut, demonstrating reduced verdict flip rates and maintained accuracy in several settings, though its effectiveness diminishes at larger model scales.
By Jianchang Su, Yiwei Yang, Wei Zhang
arXiv:2610. 29052v1 Announce Type: cross Abstract: A typed decision maps evidence, a question and a fixed set of options to a probability for every option, computed in one forward pass, by reading a language model's logits or a small head over its hidden states, instead of generating and parsing text.
By Yuchuan Weng, Chi Liu, Xin Chen, Srinivasan Manoharan
arXiv:2609.00654v1 Announce Type: new
Abstract: We describe the SciTrue team's participation in both subtasks of the NTCIR-19 SciClaimEval task~\cite{sciclaimeval}, which asks systems to verify scien...
By Qiming Bao, Ne\c{s}et \"Ozkan Tan, Siyuan Wang, Mark Gahegan
The paper "trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories" examines the limitations of outcome-only evaluation for large language model agents. Using a deterministic tool‑using support‑desk environment with a scripted oracle policy and a fault injector, the authors compare five different judging approaches—programmatic rules, outcome‑only, step‑rubric at two model sizes, and a self‑consistency ensemble—on metrics such as detection, step localisation, fault typing, calibration, and cost across 400 trajectories. The study finds that outcome‑only judges miss many silent faults and generate false positives, while step‑rubric judges achieve higher recall with no false alarms but at greater cost, and that none of the judges read the final reply, allowing fabricated promises to evade detection.
"whyItMatters":"The findings highlight that current production‑default outcome‑only evaluations can overlook critical failures in agent behavior, underscoring the need for more nuanced, step‑level judging methods to ensure reliable LLM agent performance."
By Hadi Mohammadi
arXiv:2608. 14617v1 Announce Type: cross Abstract: A recurring proposal in legal AI is to improve case-outcome prediction by fusing uncertainty tools (evidence graphs with belief propagation, sequential Bayesian odds updating, Dempster-Shafer combination, and conformal prediction) into one pipeline.
By Surya Saka
The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.
By Radhika Gaonkar
arXiv:2608. 07813v1 Announce Type: new Abstract: An LLM judge deployed inside a reasoning pipeline does not merely measure quality, it decides which answer ships.
By Yiyao Zhang, Diksha Goel, Hussain Ahmad, Shixun Huang, Jun Shen
The paper investigates whether fast, cost‑efficient decision models can classify the type of conflict among retrieved sources for a query, a task that improves response quality when the conflict type is known. Using the CONFLICTS benchmark, the authors evaluate 11 commercial decision models against Gemini LLMs, finding that most achieve 50–67% accuracy, often matching or surpassing Gemini 2.5 Flash while incurring 4–39× lower cost and 2.6–6.0× lower latency. A cascade approach that forwards uncertain cases to Gemini 3.8 Flash yields comparable performance at only 25–37% of its cost, and the resulting typed responses generally behave as expected when compared to LLM‑derived types.
By Parv Khatri, Harshul Jain, Mehak Kapoor, Tanya Sah
arXiv:2610.18298v1 Announce Type: new
Abstract: What binds large language models on auditable, hypothesis-driven decision tasks, where every candidate can be checked against the evidence? On the publ...
By Jing Liu, Yongxing Qi, Ping Yao, Ting Wu
arXiv:2610.20532v1 Announce Type: cross
Abstract: A multimodal agent must decide whether visual evidence is worth acquiring, which candidate or resolution to query, and when further interaction can n...
By Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Peng Zhang, Daren Zha, Jun Xiao
arXiv:2605.11467v2 Announce Type: replace-cross
Abstract: Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of *reasoning theater*: deliberativ...
By Swapnil Parekh, Naman Goyal