The paper argues that cross‑view correspondence, commonly used in agent evaluation and trace‑based learning, functions as a measurement intervention. Removing or altering this correspondence can create artificial sensitivity or invariance, and multiple optimal correspondences can obscure mechanism labels and learning credit. The authors propose a validity theory with two‑sided validation, all‑optima identification, and uncertainty propagation, and demonstrate through experiments that unvalidated correspondences can misattribute credit and erase harmful responses.
arXiv:2608. 05235v1 Announce Type: cross Abstract: Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings and retain the resulting trajectories to guide later decisions.
arXiv:2608. 19760v1 Announce Type: cross Abstract: Audited against causal ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals used to train LLM agents -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- identifies which steps causally matter better than chance.
The paper introduces the concept of perfect aliasing, where a truth probe that aligns truthful reporting with a task’s prescribed action cannot differentiate between the two based solely on its labels. In a binary reporting game, probes fitted on compliant contexts yield identical optimizations, while on rival contexts their labels are complementary, causing their AUROCs to sum to one across 751 cell-layer pairs. By employing randomized codebooks and mixed-context fitting, the authors demonstrate that separating prescribed output symbols from semantic action enables perfect recovery of truth, achieving an AUROC of 1.000 on rival trials for a reward-trained Gemma-2-9B policy, whereas conventional probes perform near chance.
arXiv:2609.36138v1 Announce Type: new Abstract: Before invoking external tools, an agentic LLM must select among a K-way action space: executing a call, seeking clarification, answering directly, or...
arXiv:2609.24194v1 Announce Type: new Abstract: Evaluation scores used around LLM systems -- including reward models, rerankers, and LLM judges -- can track surface form instead of the quality they c...
The paper introduces the twin‑prefix framework to evaluate how the size of the verification unit—i.e., how many actions a pre‑execution LLM monitor reviews in one call—affects its performance. By pairing each gold plan with a twin that differs by a single write and injecting a controlled error, the authors isolate the impact of review length on catch rates and false rejections. Their findings show that longer review windows increase rejection rates but do not improve discrimination, with the highest informedness occurring at one or two actions across all judges and domains.
arXiv:2607. 27069v2 Announce Type: cross Abstract: Closed yes/no spatial benchmarks can reward a correct answer even when the image adds little support beyond no-image contexts.
The paper introduces a framework for evaluating AI systems that not only checks final labels but also tracks the reasoning behind them through three core sources—grounds, norms, and authority—forming an eight-cell counterfactual judgment cube. It defines minimal source replacement sets, called judgment receipts, to explain changes in verdicts and provides certification cost bounds for black-box evaluators. The authors present ReasonBench, a benchmark with 19,520 cases, and demonstrate that while high standard accuracy can mask robustness issues, receipt accuracy reveals significant gaps in reasoning consistency across different models.
The paper introduces ontological trust, a task‑conditioned property of trajectory prefixes, and presents RGE, an online monitor that decomposes trust into Role, Goal, and Evidence. RGE uses LLMs only for structured task and step representations, while trust updates and interventions are deterministic, producing a replayable and auditable trust trajectory. Evaluated on a cross‑domain corpus, RGE outperforms rule‑, judge‑, and shield‑style baselines, achieving over 93% Drift F1 and maintaining high benign coverage.
The paper introduces Relation‑Orbit, a symmetry‑aware method for aggregating likelihoods from frozen vision‑language models to verify fine‑grained left‑right claims. It uses a closed‑form contrast that assigns eight normalized likelihoods based on reflection, inverse relation, and entity exchange, and asserts a claim only when the signed contrast exceeds a threshold chosen via Clopper‑Pearson bounds. Experiments on VSR, GQA, and LLaVA‑1.5/COCO show that Relation‑Orbit achieves higher mean test coverage at a 10% selective‑risk calibration target compared to an all‑eight Orbit‑Max baseline across multiple dataset‑backbone settings.
arXiv:2606. 29713v1 Announce Type: cross Abstract: Hallucination is the reliability bottleneck for LLM-based agents, and fact attribution verifiers are the last line of defense -- yet today's verifiers emit only opaque binary labels, leaving agents unable to self-correct and operators unable to audit.
The paper demonstrates that open‑ended Theory‑of‑Mind trackers can produce valid beliefs that are absent from finite reference sets, and that treating unmatched outputs as false can reverse model‑selection rankings. By recoding references for 259 beliefs, the authors show a dramatic drop in weighted prevalence and a reversal of strictly proper Brier risk, with similar distortions observed in a 301‑question NQ‑open DPR‑BERT pipeline. The study further reveals that 90‑96% of audited unmatched beliefs are literally true, and introduces a TriSource‑Restore method that anchors reference labels to a probability‑sampled human pilot to restore calibration and ranking integrity.