arXiv AI
Credit Without Ground Truth: Claim-Matched Evaluation of Step-Level Credit with Finite Executed Replay
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 21
Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay
arXiv:2608. 19760v1 Announce Type: cross Abstract: Audited against causal ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals used to train LLM agents -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- identifies which steps causally matter better than chance.
arXiv Machine Learning
Jun 16
Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation
arXiv:2606. 15621v1 Announce Type: new Abstract: Per-token counterfactual credit estimation asks which token in a language-model rollout caused the final answer to be right or wrong: cut the transcript at a pivot, substitute an alternative token, replay continuations, and compare outcomes.
arXiv Machine Learning
Jun 18
The Illusion of Improvement: Reject Inference Strategies in Credit Scoring
arXiv:2606. 18479v1 Announce Type: new Abstract: Reject inference methods are widely used to mitigate survival bias in credit scoring, yet their effectiveness remains poorly understood.
arXiv AI
Oct 1
Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Banking
The paper evaluates an agentic harness that combines a 27B language model, financial calculations, narrative templates, and validation checks to produce corporate and investment banking presentation decks. Using a panel of five judges, the system consistently scores higher than a baseline model that generates directly from a short prompt, with scores ranging from 20.4 to 33.6 out of 95. However, judge variability and changes in grading criteria make it challenging to discern small improvements in deck quality.
arXiv AI
Sep 25
When Does Action Credit Need Updating?
The paper investigates when historical action credit for tool‑using agents must be updated after policy changes. It introduces pairwise branch sensitivity to measure how policy updates affect action‑distinguishing branches, and proposes a first‑order anchored credit‑transport estimator along with a Decision‑Sufficient Credit Gate (DSC‑Gate) to decide whether to reuse, transport, or resample credit. Experiments show that DSC‑Gate reduces new tool steps by 39.4% with negligible impact on regret, demonstrating that many policy updates can avoid costly recomputation of action credit.
arXiv Machine Learning
Aug 27
Same-Player Verification for Account Consistency in Counter-Strike 2
The paper introduces a method for verifying whether two gameplay replays in Counter‑Strike 2 belong to the same player by extracting a behavioral fingerprint that captures crosshair control, movement‑stop‑fire coordination, economy, combat engagement, and temporal rhythm. Using 1,330 demos and 13,300 observations, the authors train a pairwise model that achieves an ROC AUC of 0.931 and 0.722 recall at 95% precision, with low‑level mechanical habits providing the strongest identity signals. Aggregating multiple demos further improves performance, raising AUC to 0.986 when ten historical demos are considered.