arXiv Machine Learning
Aug 4
Information-Theoretic Foundations for Machine Learning
arXiv:2407. 12288v5 Announce Type: replace-cross Abstract: The progress of machine learning over the past decade is undeniable.
Comparing Corrupted Constrained Learning Problems
The paper discusses the data processing inequality (DPI) in statistics, which states that a stochastically modified experiment cannot have a lower Bayes risk than the original. It shows that this classical DPI does not hold for constrained learning problems common in machine learning, where the model class is limited. The authors propose a generalized DPI that applies to constrained Bayes risks, linking it to a set containment condition on a superprediction set, and provide sufficient conditions for this containment.
Related stories
arXiv Machine Learning
Jun 8
Stability beyond Bounded Differences: Sharp Generalization Bounds under Finite $L_p$ Moments
arXiv:2606. 06855v1 Announce Type: cross Abstract: While algorithmic stability is a central tool for understanding generalization of learning algorithms, existing high-probability guarantees typically rely on uniform boundedness or sub-Gaussian/sub-Weibull tail assumptions, which can be overly restrictive for modern settings with heavy-tailed or unbounded losses.
arXiv Machine Learning
Jun 17
Does the Data Processing Inequality Reflect Practice? On the Utility of Low-Level Tasks
arXiv:2512. 21315v2 Announce Type: replace Abstract: The data processing inequality is an information-theoretic principle stating that the information content of a signal cannot be increased by processing the observations.
arXiv AI
Jul 7
Machine Unlearning via Information Theoretic Regularization
arXiv:2502. 05684v5 Announce Type: replace-cross Abstract: How can we effectively remove or ``unlearn'' undesirable information, such as specific features or the influence of individual data points, from a learning outcome while minimizing utility loss and ensuring rigorous guarantees?
arXiv Machine Learning
Sep 17
Efficient Robust Learning at the Information-Theoretic Limit
The paper presents a polynomial‑time algorithm for robustly learning Boolean concept classes with respect to a fixed distribution, achieving the optimal error rate of η + ε where η is the noise rate. It builds on Blanc’s earlier, computationally inefficient algorithm and introduces no‑regret learners to overcome the previous limitations. Additionally, the authors provide an efficient method that does not require an ERM oracle for any function class admitting sandwiching polynomials under hypercontractive distributions, including a first polynomial‑time solution for learning halfspaces with Gaussian marginals at error η + ε.
arXiv Machine Learning
2d ago
Can Domain Generalization be Guaranteed in Small-Sample Learning?
arXiv:2609.39512v1 Announce Type: new Abstract: The small-sample learning problem remains a fundamental challenge in machine learning because limited training data lead to unstable model estimation a...
arXiv Machine Learning
Sep 22
Classification with Abstention Under Class-Conditional Error Constraints
The paper investigates binary classification with abstention under separate class‑conditional error constraints, aiming to minimize abstention while keeping both error types below specified thresholds. It derives the distribution‑free minimax rate of excess abstention risk, introduces surrogate‑loss formulations for computational feasibility with models like neural networks, and provides finite‑sample guarantees for excess surrogate ambiguity risk. The authors also formulate the learning task as a constrained optimization problem, analyze its computational complexity in the convex setting, and empirically evaluate the approach against a competing method on several datasets.
arXiv Machine Learning
Aug 11
Optimal Learning Under Tsybakov Noise
arXiv:2608. 08416v1 Announce Type: new Abstract: Probably Approximately Correct (PAC) learning [Val84] is a fundamental learning model that has been extensively investigated.
arXiv AI
Jun 4
Bayes-Sufficient Representations in Supervised Learning
arXiv:2606. 04045v1 Announce Type: cross Abstract: Representation learning is often described as preserving the information in an input that is relevant for prediction.
arXiv Machine Learning
4d ago
Learning Distributionally Robust First-Order Methods for Convex Optimization
The paper introduces a distributionally robust method for learning hyperparameters of first‑order convex optimization algorithms. By minimizing a Wasserstein‑robust performance estimation problem over a dataset of problem instances, the approach interpolates between classical learning‑to‑optimize (L2O) and worst‑case PEP design. The authors solve the resulting problem with stochastic gradient descent, provide high‑probability risk bounds, and demonstrate that the learned algorithms outperform both worst‑case optimal and vanilla L2O baselines on logistic regression, LASSO, and linear programming tasks.
arXiv Machine Learning
Jul 16
Variational Inference for Evidential Deep Learning
arXiv:2605. 26477v2 Announce Type: replace Abstract: While Deep Neural Networks (DNNs) achieve remarkable performance, their tendency to produce overconfident predictions.
arXiv Statistics ML
3d ago
Constrained Classification and Policy Learning
The paper investigates the consistency of surrogate loss methods for classification and policy learning when the set of admissible classifiers is constrained, such as by interpretability or fairness requirements. It shows that hinge loss is the only surrogate that preserves consistency when constraints limit only the prediction set, but consistency can fail if constraints also restrict the functional form. The authors derive conditions guaranteeing consistency for hinge-risk-minimizing classifiers and use these results to design efficient hinge-loss-based procedures for monotone classification problems.