arXiv AI
Jun 2
Domain-Shift-Aware Conformal Prediction for Large Language Models
arXiv:2510. 05566v2 Announce Type: replace-cross Abstract: Large language models have achieved impressive performance across diverse tasks.
Bridging Domain Expertise and Generalization for Performance Estimation
arXiv:2606. 06335v1 Announce Type: new Abstract: Performance estimation under distribution shift aims to predict how a model behaves on an unlabeled test set whose distribution differs from the training data, a scenario that requires reliable indicators that can faithfully reflect model behavior without ground-truth labels.
Related stories
arXiv Machine Learning
Oct 21
Adaptive Dual-Teacher Distillation with Subnetwork Rectification for Bridging Semantic Gaps in Black-Box Domain Adaptation
The paper introduces Adaptive Dual-Teacher Distillation with Subnetwork Rectification (DDSR), a method for black-box domain adaptation that fuses predictions from a black-box source model and a vision‑language model to generate reliable pseudo‑labels for the target domain. DDSR uses adaptive prediction fusion, subnetwork regularization to prevent overfitting, iterative refinement of pseudo‑labels and prompts, and class‑wise prototypes for self‑training. Experiments on several benchmark datasets show that DDSR outperforms existing state‑of‑the‑art approaches, even those that have access to source data or source model parameters.
arXiv Machine Learning
Oct 16
INCONVENIENT: A Benchmark to Assess LLM Generalization Capabilities from Convenience Samples
INCONVENIENT is a new benchmark that evaluates how machine‑learning models, especially large language models, generalize when trained on convenience samples that bias feature–target relationships. Unlike prior distribution‑shift tests that rely on natural domain splits, INCONVENIENT explicitly perturbs these relationships while keeping the deployment population unchanged, allowing controlled study of shift severity and direction. The benchmark covers 200 tabular classification and regression datasets and produces about 15,500 shifted training distributions, revealing significant performance degradation.
arXiv Machine Learning
Jul 24
DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
arXiv:2607. 20465v1 Announce Type: new Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end.
arXiv AI
Sep 10
From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models
The paper investigates how synthetic pretraining priors used in tabular foundation models (TFMs) influence downstream performance. By reconstructing the synthetic data generators of four TFMs and comparing their generated tasks to two popular tabular benchmarks using structural descriptors, the authors measure structural coverage and normalized density. They find that some generators provide broader and denser support for benchmark tasks, and that stronger synthetic-to-benchmark support generally correlates with better model performance.
arXiv AI
Oct 5
Test-time Calibration Learning for Large Language Model Reasoning
The paper introduces Test-Time Calibration Learning (TTCL), a label‑free framework that adapts both reasoning accuracy and verbalized confidence of large language models directly on unlabeled target‑task data. TTCL generates self‑supervision signals from multiple model responses, enabling calibration without ground‑truth labels and proving theoretically as a bounded surrogate for the ideal calibration objective. Experiments on mathematical reasoning and factual question answering show consistent improvements, with base models gaining an average 40.13% in accuracy and 70.80% in ECE reduction across eight benchmarks, and further gains under domain shift.
arXiv AI
Oct 5
Safeguarding Mutual Correction in Source-Free Domain Adaptation via Cut Statistics
The paper introduces SafeCut, a method for Source-Free Domain Adaptation that uses Vision‑Language models as external knowledge. SafeCut employs a cut statistic to gauge prediction reliability, enabling a dynamic, reliability‑gated supervision between the source‑pretrained model and the ViL model. This approach selectively amplifies correct mutual corrections while suppressing error propagation, achieving state‑of‑the‑art performance on multiple SFDA benchmarks.
arXiv Statistics ML
Sep 28
Conformal Prediction under Exponential-Tilt Joint Shift
arXiv:2609.30886v1 Announce Type: new Abstract: Conformal prediction can lose coverage when the data distribution changes after deployment. We study adaptation using labeled source data and unlabeled...
arXiv AI
Oct 9
Predicting Alignment Generalization with Value Representations
The paper introduces the task of predicting alignment generalization, which measures how fine‑tuning a language model to follow a specific value affects its behavior across a broad set of other values. Through a large‑scale study of 66 values, the authors show that representations derived from model activations when applying values in context outperform textual description‑based methods, achieving a correlation of 0.45 versus 0.05. They further demonstrate that these representations can predict downstream robustness and help construct a taxonomy of LLM values based on empirical generalization dynamics.
arXiv Machine Learning
Aug 4
Learning the Pareto Frontier of Predictive Models under Distribution Shift
arXiv:2608. 00632v1 Announce Type: new Abstract: Modern machine learning pipelines increasingly rely on reusing pretrained and foundation models across downstream tasks.
arXiv AI
Sep 30
Alignment Forecasting: Predicting Misalignment From Training Data
The paper introduces Alignment Forecasting, a method for predicting whether fine‑tuning a language model on a given dataset will increase specific alignment failures such as deception or sycophancy. It presents ALIGNMENTFORECASTBENCH, a benchmark of over 5,000 forecasting questions across many models, datasets, and failure modes, and shows that a simple forecasting scaffold using an LLM’s assessment of dataset bias can outperform baseline forecasters. The authors demonstrate that filtering out high‑risk training examples identified by the forecaster can improve alignment in multiple‑choice evaluations, though benefits in open‑ended conversations remain uncertain.
arXiv Machine Learning
Oct 12
Optimizable Model Averaging for Ensembling Heterogeneous Model Outputs
Optimizable Model Averaging (OMA) is a black‑box method that assigns nonnegative weights to each candidate model’s predictive probabilities, keeping all model parameters frozen. By minimizing regularized predictive entropy, OMA learns a single scalar coefficient per model to average outputs from heterogeneous architectures. The authors show that appropriate regularization reduces expected calibration error and improves the trade‑off between performance and calibration compared to simple output averaging.