arXiv Machine Learning

When Does Dynamic Ensembling Pay Off? Diagnosing Regionwise Gains in Regression under Distribution Shift

The paper introduces “ℝD_{CF5}”, a probe‑based estimator that predicts the region‑wise gain of a dynamic ensemble over the best static blend in regression tasks under distribution shift. Across 12 benchmark dataset‑shift pairs, the estimator achieves a Spearman correlation of +0.98 with actual test gains, outperforming alternative diagnostics. The authors also present a Probe‑Validated Ensemble Selector that chooses between a static affine stacker and dynamic realizers, demonstrating risk reductions of up to 16% in prospective deployments.

Hugging Face Trending Papers

Jun 10

Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness

End-to-end task-success is the dominant way to evaluate LLM agents, but one aggregate number tells you that an agent regressed, not where. We present layer-isolated evaluation: a deployed ordering agent is decomposed into a fixed taxonomy of layers (ontology, intent, routing, decomposition, escalation, safety, memory, and cross-cutting envelope/defense), each exercised by its own assertion slice in a deterministic, no-LLM "pure" mode.

llms agents safety

arXiv AI

Jun 11

Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness

arXiv:2606. 11686v1 Announce Type: cross Abstract: End-to-end task-success is the dominant way to evaluate LLM agents, but one aggregate number tells you that an agent regressed, not where.

By Sawyer Zhang, Alexander Wang, Sophie Lei

llms agents safety

arXiv Machine Learning

Aug 14

Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection

arXiv:2608. 12652v1 Announce Type: cross Abstract: Benchmark contamination is diagnosed today with n-gram overlap, with likelihood-based membership inference, or with canary strings, and each needs something usually unavailable: the training corpus, a well-chosen test statistic, or foresight at dataset release.

By Florian Braun

llms benchmarks

arXiv Machine Learning

Jul 8

Heckman-Corrected Epistemic Uncertainty: Selection on Unobservables Defeats Importance Weighting

arXiv:2607. 05806v1 Announce Type: new Abstract: Training data for machine learning is routinely collected by a selection process the model never sees: loans are observed only when granted, outcomes only when a test was ordered.

By Gunner Levi Howe

safety

arXiv Machine Learning

Aug 11

Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk

arXiv:2608. 08126v1 Announce Type: new Abstract: Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters, in tension with supervisory expectations that adverse decisions be explainable.

By Gregorius Reynaldi Pratama, Kuo-Kun Tseng

llms

arXiv Machine Learning

Jun 24

Anticipating the Optimism Gap: Predicting Distribution-Shift Degradation of RF-Impairment Detectors from In-Distribution Statistics

arXiv:2606. 22054v2 Announce Type: replace-cross Abstract: Detectors for GNSS radio-frequency impairments (jamming, spoofing, multipath) are usually reported with a single AUC measured on the distribution they were tuned on.

By Chakshu Baweja

arXiv AI

Jul 21

First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

arXiv:2607. 16821v1 Announce Type: cross Abstract: Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updates should compose with controlled interference, useful tangent structure should be stable and possible to estimate, and weight edits should have counterparts in representation space.

By Irina Piontkovskaia, Sergey Nikolenko

llms fine-tuning

arXiv AI

Aug 18

Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusion

arXiv:2608. 14617v1 Announce Type: cross Abstract: A recurring proposal in legal AI is to improve case-outcome prediction by fusing uncertainty tools (evidence graphs with belief propagation, sequential Bayesian odds updating, Dempster-Shafer combination, and conformal prediction) into one pipeline.

By Surya Saka

llms

arXiv AI

Jun 16

Heteroskedastic Signals in Budgeted LLM Verification: Structural Heterogeneity Limits Optimization Gains

arXiv:2606. 15841v1 Announce Type: new Abstract: Large language model (LLM) systems increasingly use uncertainty signals to allocate limited computation across verification, test-time scaling, tool execution, and other selective-compute decisions.

By Jinlong Yang

llms

arXiv Machine Learning

Aug 14

When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide

arXiv:2608. 12489v1 Announce Type: new Abstract: Organizations decide whom to treat under a budget and want to know what a targeting rule would have earned before deploying it.

By Binshuang Li

benchmarks safety

arXiv Machine Learning

Jun 9

A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control

arXiv:2606. 08517v1 Announce Type: new Abstract: Selective predictors answer on confident inputs and abstain elsewhere; deploying one safely needs a single finite-sample certificate that simultaneously upper-bounds the selected risk, lower-bounds the acceptance probability $\pacc$ above a floor $\pmin$, and lower-bounds the deployment utility.

By Xiaoli Yu, Jiamiao Liu

arXiv Machine Learning

Jul 30

Dynamic Parameterization Is Not Dynamic Inference

arXiv:2607. 26192v1 Announce Type: new Abstract: Input-dependent controller coefficients are often treated as evidence of dynamic inference or computational savings.

By Zongfei Li, Yuan-yih Shang, Guozhong Luo

llms

Related stories