The paper investigates how to combine two commonly used classification performance metrics into a single ranking that balances them optimally. It shows that all such metrics can be expressed as ranking scores parameterized by the relative importance of true/false positives and negatives, and derives an exact analytical solution for the optimal tradeoff between any pair of these scores when class priors are equal. The authors also explore experimentally how the solution behaves when performance follows certain distributions with fixed priors.
arXiv:2604. 01506v2 Announce Type: replace Abstract: Long-tailed classification, where a small number of frequent classes dominate many rare ones, remains challenging because models systematically favor frequent classes at inference time.
arXiv:2605. 05189v2 Announce Type: replace-cross Abstract: How many key-value associations can a $d\times d$ linear memory store?
The paper demonstrates that common binary classification metrics—Matthews' correlation coefficient, Cohen's κ, the F-score, and the Jaccard similarity—are not robust to extreme class imbalance, as the Bayes classifier’s true positive rate tends to zero when the minority class proportion vanishes. To address this, the authors propose robustified versions of these metrics that include a tuning parameter, ensuring that the Bayes-optimal classifier’s threshold remains bounded and its true positive rate stays above zero even in highly imbalanced scenarios. The study provides theoretical bounds, simulation results, and practical guidance on applying these robust metrics to real data, such as a credit‑default dataset, and discusses their relationship to ROC and precision‑recall curves.
The paper introduces the Adaptive Margin Ordinal Loss (AMOL), a new loss function designed to reduce the tendency of neural networks to predict center classes in ordinal classification tasks—a problem called center‑class hedging. AMOL applies a multiplicative weight to per‑class loss terms that is large only when a candidate class is near the center while the true label is far from it, thereby discouraging hedging. The authors also propose the Center‑Hedging Rate (CHR) metric to quantify this failure mode and demonstrate that AMOL achieves state‑of‑the‑art Quadratic Weighted Kappa scores on four benchmarks, with an asymmetric variant eliminating hedging on the Abalone dataset.
The paper introduces Rank‑Variance Scores (TRV) as a new metric for importance sampling and data pruning in the presence of label contamination. It shows that traditional loss‑based methods fail because corrupted examples often have high loss, and proves that a useful score must be non‑monotonic in loss. By ranking examples across multiple independently trained proxy models, TRV captures both between‑proxy disagreement and within‑proxy variance, enabling Disagreement‑Regularized Importance Sampling (DR‑IS) and TRV‑based pruning to outperform random selection on noisy CIFAR‑10/100 datasets.
arXiv:2606. 27997v1 Announce Type: new Abstract: Benchmarks of machine learning models often include many datasets, making evaluation expensive.
The paper introduces a Bayesian Bradley‑Terry model (BBTM) that employs Polya‑Gamma data augmentation for efficient Gibbs sampling and quantifies uncertainty in latent abilities and ranks. Three active‑learning strategies based on the knowledge‑gradient principle are proposed, using Kendall’s tau distance, posterior rank entropy, and posterior separation as acquisition functions, with approximate one‑step updates to reduce computational cost. The authors also present Posterior Separation Reliability (PSR) as an internal indicator of ranking reliability, and demonstrate through synthetic experiments that BBTM outperforms preference‑based Bayesian comparative judgement, with the entropy‑based strategy excelling at low‑to‑moderate noise and the posterior‑separation strategy at high noise levels.
Comparative judgement uses pairwise comparisons to rank items, but recovering accurate rankings from a limited number of judgements depends on both the inference method and the pairs selected. We deve...
arXiv:2607. 01715v1 Announce Type: new Abstract: Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level.
arXiv:2610.28025v1 Announce Type: cross Abstract: This paper studies the problem of aggregation in multiclass classification, generalizing existing results for binary classification. Given a finite d...
arXiv:2505. 23437v2 Announce Type: replace-cross Abstract: Ranking systems influence decision-making in high-stakes domains like health, education, and employment, where they can have substantial economic and social impacts.
arXiv:2608. 02081v1 Announce Type: new Abstract: In this paper, we study prediction problems for paired comparison data, for example, predicting the win probability between two unmatched players and ranking all the players according to the order of their strengths by using win probability data between two matched players.