GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?
arXiv:2606. 22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence?
The paper introduces a risk‑controlled framework for using large language models (LLMs) as judges in tasks without reference answers. By calibrating uncertainty thresholds on a held‑out set, the method ensures that the false discovery rate of accepted verdicts stays below a user‑specified level α with high probability, using finite‑sample Clopper–Pearson intervals. When the parametric judge lacks confidence, the instance is routed to a retrieval‑augmented mode with a second calibrated threshold, preserving the error guarantee while achieving higher coverage than single‑mode baselines.
arXiv:2607. 12650v1 Announce Type: cross Abstract: Tool access alone does not make LLM empirical reasoning governable: accepted outputs need not descend from attested evidence, and accepted deductions need not hold up under formal scrutiny.
arXiv:2608. 05235v1 Announce Type: cross Abstract: Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings and retain the resulting trajectories to guide later decisions.
arXiv:2607. 20827v1 Announce Type: new Abstract: LLM agents choose tools and arguments from context that mixes user requests, tool outputs, retrieved records, memory, and untrusted text.
arXiv:2610.34826v1 Announce Type: new Abstract: Decision models enable software to select from predefined actions using natural language inputs. However, the external evidence that supplies facts for...
arXiv:2607. 25364v1 Announce Type: new Abstract: Tool-using agents expose structured calls but commonly attach free-form rationales.
The paper introduces a framework for evaluating AI systems that not only checks final labels but also tracks the reasoning behind them through three core sources—grounds, norms, and authority—forming an eight-cell counterfactual judgment cube. It defines minimal source replacement sets, called judgment receipts, to explain changes in verdicts and provides certification cost bounds for black-box evaluators. The authors present ReasonBench, a benchmark with 19,520 cases, and demonstrate that while high standard accuracy can mask robustness issues, receipt accuracy reveals significant gaps in reasoning consistency across different models.
The paper investigates whether activation interventions that guide a language model toward a target answer also enable the model to use the intended condition in other computations. Using Qwen3.5‑27B on tool‑use tasks, the authors find that residual‑writing variants improve target‑selection accuracy but these gains vanish when familiar facts and identifiers are recombined, and they do not translate into better independent judgments about eligibility or selection. The study demonstrates a clear separation between target‑selection control, condition‑sensitive judgments, and complete‑action correctness, highlighting the importance of evaluating how edited states influence downstream reasoning rather than just the final answer.
arXiv:2603. 05167v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning, yet it remains unclear whether they can reliably assess process faithfulness rather than merely answer plausibility.
The paper introduces SARA, a framework that separates action induction from runtime authorization in tool‑augmented LLM agents. By treating these as distinct roles, SARA uses an Action Probe to record action provenance and only authorizes tool calls that align with the user objective and past successful executions. Experiments on AgentDojo and AgentDyn show that SARA reduces action‑to‑side‑effect risk to below 0.63% while preserving task performance.
Operational reporting for high-performance computing (HPC) facilities requires combining information from monitoring services, schedulers, incident and maintenance records, storage measurements, perio...