The paper introduces EDGE, a grouped calibration test for logistic regression that remains robust when a few records are corrupted. EDGE projects grouped residuals onto a small basis of calibration shapes, keeping degrees of freedom independent of group count, which allows fine‑grained power or coarse protection against outliers. In simulations and external validation, EDGE matched or exceeded the power of existing tests while maintaining false‑alarm rates below 10% even with corrupted data.
arXiv:2607. 28608v1 Announce Type: new Abstract: Clinical risk models routinely achieve strong aggregate performance while producing materially different error rates across patient subgroups.
arXiv:2607. 18828v1 Announce Type: new Abstract: Readiness stress-testing of medical AI has focused on closed-ended and multimodal benchmarks.
arXiv:2608.21262v1 Announce Type: cross Abstract: Many machine-learning systems set a threshold at a quantile of a calibration set: conformal predictors that promise 90% coverage by drawing their cut...
DeepGOF-1 is a neural goodness‑of‑fit test for logistic regression that maps residuals over covariate ranks, scores them with a frozen convolutional network, and calibrates the score via parametric bootstrap. It achieves first‑order valid level, offers explicit local power, and outperforms traditional chi‑square and projection tests in simulations and real data, revealing missed interactions and guiding model repairs. In the SUPPORT study, it identified a U‑shaped risk pattern for blood pressure and respiratory rate that standard calibration curves obscured, and it ran in 13 seconds versus hours for competing methods.
The paper presents a closed‑form estimator for the contamination correlation between anchors and judges under a single‑common‑factor model, requiring at least two judges and two anchors. It introduces a diagnostic battery—including judge‑covariance dispersion, over‑identification tests, a family‑block test, bootstrap confidence intervals, and a weak‑identification screen—to validate the estimator and detect violations. The authors also discuss identification limits for ordinal data and report that real panels have not yet passed the model‑adequacy pre‑test, while simulation studies confirm the estimator’s performance.
arXiv:2604. 17359v2 Announce Type: replace-cross Abstract: Language models asked to simulate psychiatric patients produce cases that survive inspection one at a time and populations that match no real one.
The paper examines how to quantify the time AI tools save, contrasting a simple stopwatch approach with a more nuanced ledger view that accounts for errors introduced. It introduces model‑free metrics—time saved, excess escaped defects, and a break‑even defect cost—and applies them to a baseline checking model, randomized studies, and a replication of METR’s developer data. The author also reports on a small experiment with AI reviewers, showing that reduced effort can lead to faster but more error‑prone outcomes.
The paper introduces two new procedures for robust clusterwise regression. First, a group‑recovery step that scans discarded observations for a line, tests whether nearby points form a peak, and restores the line as a group when the likelihood improves by a BIC‑like margin. Second, an exact‑subsample flagging (ESF) method that flags outliers without a fixed trimming level by solving the least‑squares problem on small subsamples and using subsample size and a flag cap as implicit controls.
The study evaluates whether fault localization improves test‑guided code repair by comparing three approaches—blind whole‑solution resampling, spectrum‑based localized infilling, and random‑span infilling—across multiple large language models and benchmarks. Results show that localization is rarely available (only 9.0% of failing candidates), and when it is, localized infilling performs worse than blind resampling, with only suggestive evidence of a benefit over random spans. The findings suggest that targeted edits may not provide a consistent advantage over broader, untargeted repair attempts in current large‑model settings.
arXiv:2609.25052v1 Announce Type: new Abstract: "An agent that writes its conclusions into a store it later retrieves from closes a loop usually reported as one-way contamination. Taking the loop to...
The paper introduces a new taxonomy for benchmark contamination that categorizes leakage by the mitigation it defeats—direct, derivative, temporal, distributional, and acquired—covering both training‑time and evaluation‑time scenarios. It proposes a four‑field disclosure protocol to record contamination status alongside benchmark scores, and provides a JSON schema, validator, and examples. An empirical study of 41 documents using a pre‑registered instrument shows limited reporting of contamination types and variable reliability, highlighting gaps in current disclosure practices.
arXiv:2608. 12652v1 Announce Type: cross Abstract: Benchmark contamination is diagnosed today with n-gram overlap, with likelihood-based membership inference, or with canary strings, and each needs something usually unavailable: the training corpus, a well-chosen test statistic, or foresight at dataset release.