PENEX: AdaBoost-Inspired Neural Network Regularization

Bernhard Sch\"olkopf; Klaus-Rudolf Kladny; Michael Muehlebach

arxiv: 2510.02107 · v4 · submitted 2025-10-02 · 💻 cs.LG

PENEX: AdaBoost-Inspired Neural Network Regularization

Klaus-Rudolf Kladny , Bernhard Sch\"olkopf , Michael Muehlebach This is my paper

Pith reviewed 2026-05-18 10:17 UTC · model grok-4.3

classification 💻 cs.LG

keywords PENEXexponential lossneural network regularizationAdaBoostmargin maximizationgeneralization boundslow-data regimesmulti-class classification

0 comments

The pith

A reformulated exponential loss allows neural networks to increase data margins via first-order optimization, supporting generalization bounds and stronger low-data performance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents PENEX, a penalized version of the multi-class exponential loss drawn from AdaBoost. This formulation penalizes misclassified points more heavily than cross-entropy while remaining compatible with gradient-based training of neural networks. The authors show that PENEX increases margins around training points, which directly yields a generalization bound. Experiments on vision and language tasks demonstrate that the approach improves test performance in low-data regimes and matches or exceeds standard regularizers without extra computational overhead.

Core claim

PENEX is a penalized formulation of the multi-class exponential loss that preserves the margin-increasing behavior of the classical AdaBoost exponential loss. It is amenable to first-order optimization and can therefore serve as a training objective for neural networks. When optimized this way, the loss produces larger margins on correctly classified points, which translate into a generalization bound. Empirical results across computer vision and language benchmarks confirm improved generalization in low-data settings at costs comparable to existing regularizers.

What carries the argument

PENEX, the penalized multi-class exponential loss that modifies the standard exponential loss to enable first-order optimization while retaining margin-maximization effects.

If this is right

Neural networks can be trained with first-order methods to achieve the margin growth previously limited to boosting algorithms.
Generalization bounds follow directly from the observed margin increases under PENEX.
Low-data performance on vision and language tasks reaches or exceeds levels of dropout, weight decay, and other standard regularizers.
The exponential loss becomes a practical objective for deep networks rather than being restricted to sequential boosting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Similar penalized reformulations could be applied to other boosting losses to adapt them for end-to-end neural training.
The margin-growth mechanism may interact productively with modern architectures such as transformers when data is scarce.
Extending PENEX to semi-supervised or self-supervised settings could test whether the margin benefit generalizes beyond fully supervised low-data regimes.

Load-bearing premise

The penalized formulation of the multi-class exponential loss preserves the margin-increasing and generalization properties of the classical AdaBoost exponential loss while remaining amenable to first-order optimization in neural network training.

What would settle it

Training a convolutional network on a low-data CIFAR-10 subset with PENEX and finding that the minimum margin over training points is no larger than under cross-entropy, or that test accuracy does not improve relative to the baseline regularizer.

Figures

Figures reproduced from arXiv: 2510.02107 by Bernhard Sch\"olkopf, Klaus-Rudolf Kladny, Michael Muehlebach.

**Figure 1.** Figure 1: Gradient Descent on PENEX as a Form of Implicit AdaBoost. AdaBoost (left) builds a strong learner fM(x) (purple) by sequentially fitting weak learners such as decision stumps (orange) and linearly combining them. Gradient descent itself (right) can be thought of as an implicit form of boosting where weak learners correspond to J(x)∆θm (orange), parameterized by parameter increments ∆θm. Combining many grad… view at source ↗

**Figure 2.** Figure 2: Comparison of Margins. Neural networks trained with PENEX (center) tend to implicitly maximize margins (here, geometric margins, indicated for an example point in green), in a similar way to support vector machines (right), here trained with a RBF kernel (Sch¨olkopf & Smola, 2002, p. 46). Training neural networks via cross-entropy loss (left), in contrast, typically leads to smaller margins. • We introduce… view at source ↗

**Figure 3.** Figure 3: CE vs. PENEX. We consider the binary case (K = 2) with f (2)(x) ≡ 0, for a single x and y = 1. PENEX penalizes errors more than cross-entropy. where GPENEX is the set of functions given in Sec. A.3. The following proposition shows that a small gradient descent step on PENEX indeed does solve (10): Proposition 2.3. In the limit η → 0, for some GPENEX and ρ > 0, the solution of (10) converges to the negativ… view at source ↗

**Figure 4.** Figure 4: Performance Analysis on CIFAR-100. Larger means better. Results are computed from validation data. All hyperparameters have been tuned individually. PENEX (thick red) is an effective regularizer with often better generalization than other common regularization techniques (thin), and shows no signs of “overfitting” like cross-entropy training (orange). Hyperparameter Tuning. Each method’s hyperparameters… view at source ↗

**Figure 5.** Figure 5: All Validation Curves. Larger means better. Validation curves over 200 epochs for all experiments, similar to [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗

**Figure 6.** Figure 6: Ablation Studies on CIFAR-10. Larger means better. a) Optimizing the exponential loss without any constraint or with the constraint encoded into the model architectures break down early on during training (no results are shown for the hard constraint method starting at around epoch 80 because of NaN model outputs). b) While constrained optimization algorithms based on CONEX (2) are trainable in principle, … view at source ↗

**Figure 7.** Figure 7: Affects of the Sensitivity Parameter on Validation Curves. Smaller α typically lead to faster convergence, but worse generalization toward the end of the training duration (as can be seen more clearly for CIFAR-100). In addition, for very large α = 3.2, training curves tend to become less stable [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗

read the original abstract

AdaBoost sequentially fits so-called weak learners to minimize an exponential loss, which penalizes misclassified data points more severely than other loss functions like cross-entropy. Paradoxically, AdaBoost generalizes well in practice as the number of weak learners grows. In the present work, we introduce Penalized Exponential Loss (PENEX), a new formulation of the multi-class exponential loss that is theoretically grounded and, in contrast to the existing formulation, amenable to optimization via first-order methods, making it a practical objective for training neural networks. We demonstrate that PENEX effectively increases margins of data points, which can be translated into a generalization bound. Empirically, across computer vision and language tasks, PENEX improves neural network generalization in low-data regimes, matching and in some settings outperforming established regularizers at comparable computational cost. Our results highlight the potential of the exponential loss beyond its application in AdaBoost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

PENEX adapts the exponential loss for gradient-based NN training and shows low-data gains, but the theory and controls need checking.

read the letter

The main thing to know is that PENEX adds a penalty to the multi-class exponential loss so it can be trained end-to-end with first-order methods in neural networks, unlike the standard formulation that does not play well with backprop. They report that this version still pushes up margins and delivers better generalization than common regularizers on vision and language tasks when data is scarce, at roughly the same cost. That empirical angle is the clearest contribution here. The paper does a reasonable job framing the idea as a bridge from AdaBoost-style losses to modern deep learning and backs it with concrete task results. The soft spots sit in the theory and the experiments. The abstract ties the generalization bound to margin growth, but without the actual equations it is hard to tell whether the bound is independent or largely circular with the fitted margins. On the runs, more ablations on penalty strength and tighter baseline comparisons would help rule out simple hyperparameter effects. This work is aimed at people who tune losses or regularizers for small-data regimes rather than core theory builders. A reader looking for a practical tweak to try on their own models would get something out of the numbers. It is solid enough on the idea and the claims to deserve a full referee process instead of a desk reject.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Penalized Exponential Loss (PENEX), a reformulation of the multi-class exponential loss that incorporates a penalty term. It claims this version remains amenable to first-order optimization (unlike the classical formulation), increases margins of data points, and yields a generalization bound. Empirically, PENEX is shown to improve neural network generalization in low-data regimes on computer vision and language tasks, matching or outperforming standard regularizers at comparable cost.

Significance. If the central theoretical claim holds—that the penalty preserves the margin-maximizing behavior of the exponential loss while enabling gradient-based training—PENEX would provide a principled, AdaBoost-inspired regularizer for deep networks in data-scarce settings. The empirical results, if robust, would demonstrate practical utility beyond boosting and motivate further loss-function designs that import classical margin theory into neural training.

major comments (2)

[Theoretical analysis section] The weakest assumption is that the penalized multi-class exponential loss preserves the margin-increasing property and allows the same generalization bound as the classical AdaBoost exponential loss. The manuscript must supply the explicit derivation (including how the penalty term affects the margin definition and whether the bound remains independent of fitted quantities) to confirm this preservation; without it the translation from margin increase to bound cannot be verified.
[Experimental results section] Empirical claims rest on low-data regimes, yet the precise definition of these regimes (sample counts per class, train/test splits) and the controls for optimizer hyperparameters and computational cost are not stated with sufficient detail to rule out confounding factors in the reported gains over baselines.

minor comments (2)

[Abstract] The abstract states that PENEX is 'theoretically grounded' but does not preview the key step linking the penalty to margin growth; a single sentence summarizing this link would improve readability.
[Method section] Notation for the penalty term and its hyperparameter should be introduced consistently in the loss definition to avoid ambiguity when comparing to the unpenalized multi-class exponential loss.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment below and outline the revisions planned for the next version of the manuscript.

read point-by-point responses

Referee: [Theoretical analysis section] The weakest assumption is that the penalized multi-class exponential loss preserves the margin-increasing property and allows the same generalization bound as the classical AdaBoost exponential loss. The manuscript must supply the explicit derivation (including how the penalty term affects the margin definition and whether the bound remains independent of fitted quantities) to confirm this preservation; without it the translation from margin increase to bound cannot be verified.

Authors: We agree that an explicit derivation is required to rigorously verify preservation of the margin-increasing property and the generalization bound. In the revised manuscript we will expand the Theoretical analysis section with a complete step-by-step derivation. This will show precisely how the added penalty term modifies the margin definition while retaining the key properties that allow the bound to remain independent of fitted quantities, thereby confirming the translation from margin increase to generalization guarantee. revision: yes
Referee: [Experimental results section] Empirical claims rest on low-data regimes, yet the precise definition of these regimes (sample counts per class, train/test splits) and the controls for optimizer hyperparameters and computational cost are not stated with sufficient detail to rule out confounding factors in the reported gains over baselines.

Authors: We concur that additional experimental detail is necessary to strengthen the empirical claims. In the revised Experimental results section we will specify the exact sample counts per class used to define the low-data regimes, provide full train/test split information, and document all optimizer hyperparameters together with computational-cost controls. These additions will allow readers to assess the fairness of the comparisons and rule out confounding factors. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper defines PENEX as a penalized multi-class exponential loss explicitly constructed to be amenable to first-order optimization while preserving margin-increasing behavior from classical AdaBoost. The generalization bound is presented as following from the demonstrated margin increase rather than being fitted or self-referential. No equations reduce a prediction to a fitted parameter by construction, no load-bearing self-citation chain is invoked for uniqueness, and the empirical results on CV/language tasks provide independent validation. The derivation chain remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review performed on abstract only; no explicit free parameters, axioms, or invented entities are stated in the provided text.

pith-pipeline@v0.9.0 · 5689 in / 971 out tokens · 25434 ms · 2026-05-18T10:17:22.929781+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We introduce Penalized Exponential Loss (PENEX) ... L_PENEX(f;α,ρ) := L_EX(f;α) + ρ Ê[SE(x)] ... margin m_f(x,y) := f^(y)(x) - max_{j≠y} f^(j)(x)
IndisputableMonolith/Foundation/AlphaCoordinateFixation.lean J_uniquely_calibrated_via_higher_derivative unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

PENEX effectively increases margins of data points, which can be translated into a generalization bound

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · 1 internal anchor

[1]

1 Christopher M. Bishop. Regularization and Complex- ity Control in Feed-forward Networks.International Conference on Artificial Neural Networks, 1995. 1 Christopher M. Bishop.Pattern Recognition and Ma- chine Learning. Springer, 2006. 1 Leo Breiman. Arcing Classifiers.Annals of Statistics, 26:123–40, 1996a. 1 Leo Breiman. Bias, Variance, and Arcing Class...

work page 1995
[2]

1 Glenn W. Brier. Verification of Forecasts Expressed in Terms of Probability.Monthly Weather Review, 78 (1):1–3, 1950. 6 Corinna Cortes and Vladimir Vapnik. Support-Vector Networks.Machine Learning, 20(3):273–297, 1995. 2 Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InConferen...

work page doi:10.5281/zenodo.4106649 1950
[3]

Deep Learning, volume 1

6 Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning, volume 1. MIT Press, 2016. 1 Derek Greene and P´ adraig Cunningham. Practical So- lutions to the Problem of Diagonal Dominance in Kernel Document Clustering. InInternational Con- ference on Machine Learning, pp. 377–384, 2006. 6 Adam J. Grove and Dale Schuurmans. Boosting in the limit: M...

work page 2016
[4]

Wein- berger

5 Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. On Calibration of Modern Neural Networks. InInternational Conference on Machine Learning, pp. 1321–1330, 2017. 6 Trevor Hastie, Robert Tibshirani, and Jerome Fried- man.The Elements of Statistical Learning. Springer, 2017. 1, 5 PENEX: AdaBoost-Inspired Neural Network Regularization Magnus R. H...

work page arXiv 2017
[5]

RoBERTa: A Robustly Optimized BERT Pretraining Approach

6 Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo- Aron Weis, Timo Gaiser, Alexander Marx, Nektar- ios A. Valous, Dyke Ferber, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study.PLoS medicine, 16(1), 2019. 6 Diederik P. Kingma and Jimmy Ba....

work page internal anchor Pith review Pith/arXiv arXiv 2019
[6]

Decoupled Weight Decay Regularization.International Conference on Learning Representations, 2019

6 Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization.International Conference on Learning Representations, 2019. 23 Shen-Huan Lyu, Lu Wang, and Zhi-Hua Zhou. Im- proving generalization of deep neural networks by leveraging margin distribution.Neural Networks, 151:48–60, 2022. 6 David J.C. MacKay.Information Theory, Inference, and Lear...

work page arXiv 2019
[7]

6 J. R. Quinlan. Boosting First-Order Learning.Inter- national Workshop on Algorithmic Learning Theory, pp. 143–155, 1996. 7 Gunnar R¨ atsch, Takashi Onoda, and K.-R. M¨ uller. Soft Margins for AdaBoost.Machine Learning, 42: 287–320, 2001. 1, 7 Saharon Rosset, Ji Zhu, and Trevor Hastie. Margin Maximizing Loss Functions.Advances in Neural In- formation Pro...

work page 1996
[8]

Dropout: A Simple Way to Prevent Neural Networks from Overfitting.The Journal of Machine Learning Re- search, 15:1929–1958, 2014

5 Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting.The Journal of Machine Learning Re- search, 15:1929–1958, 2014. 1 Georg Still.Lectures on Parametric Optimization: An Introduction. Preprint, University of Twente, 2018. 16 Ke Sun, Zhanxing Zhu,...

work page 1929
[9]

Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance.arXiv preprint arXiv:2304.11127, 2023

1 Ilya O Tolstikhin, Sylvain Gelly, Olivier Bous- quet, Carl-Johann Simon-Gabriel, and Bernhard Sch¨ olkopf. AdaGAN: Boosting Generative Mod- els.Advances in Neural Information Processing Sys- tems, 30, 2017. 6 Leslie G. Valiant. A Theory of the Learnable.Com- munications of the ACM, 27(11):1134–1142, 1984. 1 Vladimir Vapnik.The Nature of Statistical Lear...

work page arXiv 2017
[10]

Multi-class AdaBoost.Statistics and Its Interface, 2:349–360, 2009

7 Ji Zhu, Hui Zou, Saharon Rosset, and Trevor Hastie. Multi-class AdaBoost.Statistics and Its Interface, 2:349–360, 2009. 2, 3 Checklist

work page 2009
[11]

Yes (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm

For all models and algorithms presented, check if you include: (a) A clear description of the mathematical set- ting, assumptions, algorithm, and/or model. Yes (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm. Not Applicable (c) (Optional) Anonymized source code, with specification of all dependencies, including...

work page
[12]

Yes (b) Complete proofs of all theoretical results

For any theoretical claim, check if you include: (a) Statements of the full set of assumptions of all theoretical results. Yes (b) Complete proofs of all theoretical results. Yes (c) Clear explanations of any assumptions. Yes

work page
[13]

Yes (b) All the training details (e.g., data splits, hy- perparameters, how they were chosen)

For all figures and tables that present empirical results, check if you include: (a) The code, data, and instructions needed to reproduce the main experimental results (ei- ther in the supplemental material or as a URL). Yes (b) All the training details (e.g., data splits, hy- perparameters, how they were chosen). Yes (c) A clear definition of the specifi...

work page
[14]

Yes (b) The license information of the assets, if ap- plicable

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include: (a) Citations of the creator If your work uses ex- isting assets. Yes (b) The license information of the assets, if ap- plicable. Yes (c) New assets either in the supplemental mate- rial or as a URL, if applicable. Yes (d) Information about ...

work page
[15]

Not Applicable (b) Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable

If you used crowdsourcing or conducted research with human subjects, check if you include: (a) The full text of instructions given to partici- pants and screenshots. Not Applicable (b) Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable. Not Applica- ble (c) The estimated hourly wage paid to ...

work page
[16]

We show that PENEX is Fisher consistent, conditionally on anyxand for anyρ >0

work page
[17]

""Exponential loss on the true-class score fy(x)

We show that PENEX is also unconditionally Fisher consistent. Step 1The population-level equivalent of (3), conditionally onx, is Ly|x PENEX(f(x)) :=E y|x h exp n −αf (y)(x) oi +ρ KX j=1 exp n f(j)(x) o .(11) We note that (11) is strictly convex inf(x) and differentiable. Hence, the unique minimum of (11), calledf ∗(x), is characterized by its gradient be...

work page 2018

[1] [1]

1 Christopher M. Bishop. Regularization and Complex- ity Control in Feed-forward Networks.International Conference on Artificial Neural Networks, 1995. 1 Christopher M. Bishop.Pattern Recognition and Ma- chine Learning. Springer, 2006. 1 Leo Breiman. Arcing Classifiers.Annals of Statistics, 26:123–40, 1996a. 1 Leo Breiman. Bias, Variance, and Arcing Class...

work page 1995

[2] [2]

1 Glenn W. Brier. Verification of Forecasts Expressed in Terms of Probability.Monthly Weather Review, 78 (1):1–3, 1950. 6 Corinna Cortes and Vladimir Vapnik. Support-Vector Networks.Machine Learning, 20(3):273–297, 1995. 2 Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InConferen...

work page doi:10.5281/zenodo.4106649 1950

[3] [3]

Deep Learning, volume 1

6 Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning, volume 1. MIT Press, 2016. 1 Derek Greene and P´ adraig Cunningham. Practical So- lutions to the Problem of Diagonal Dominance in Kernel Document Clustering. InInternational Con- ference on Machine Learning, pp. 377–384, 2006. 6 Adam J. Grove and Dale Schuurmans. Boosting in the limit: M...

work page 2016

[4] [4]

Wein- berger

5 Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. On Calibration of Modern Neural Networks. InInternational Conference on Machine Learning, pp. 1321–1330, 2017. 6 Trevor Hastie, Robert Tibshirani, and Jerome Fried- man.The Elements of Statistical Learning. Springer, 2017. 1, 5 PENEX: AdaBoost-Inspired Neural Network Regularization Magnus R. H...

work page arXiv 2017

[5] [5]

RoBERTa: A Robustly Optimized BERT Pretraining Approach

6 Jakob Nikolas Kather, Johannes Krisam, Pornpimol Charoentong, Tom Luedde, Esther Herpel, Cleo- Aron Weis, Timo Gaiser, Alexander Marx, Nektar- ios A. Valous, Dyke Ferber, et al. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study.PLoS medicine, 16(1), 2019. 6 Diederik P. Kingma and Jimmy Ba....

work page internal anchor Pith review Pith/arXiv arXiv 2019

[6] [6]

Decoupled Weight Decay Regularization.International Conference on Learning Representations, 2019

6 Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization.International Conference on Learning Representations, 2019. 23 Shen-Huan Lyu, Lu Wang, and Zhi-Hua Zhou. Im- proving generalization of deep neural networks by leveraging margin distribution.Neural Networks, 151:48–60, 2022. 6 David J.C. MacKay.Information Theory, Inference, and Lear...

work page arXiv 2019

[7] [7]

6 J. R. Quinlan. Boosting First-Order Learning.Inter- national Workshop on Algorithmic Learning Theory, pp. 143–155, 1996. 7 Gunnar R¨ atsch, Takashi Onoda, and K.-R. M¨ uller. Soft Margins for AdaBoost.Machine Learning, 42: 287–320, 2001. 1, 7 Saharon Rosset, Ji Zhu, and Trevor Hastie. Margin Maximizing Loss Functions.Advances in Neural In- formation Pro...

work page 1996

[8] [8]

Dropout: A Simple Way to Prevent Neural Networks from Overfitting.The Journal of Machine Learning Re- search, 15:1929–1958, 2014

5 Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting.The Journal of Machine Learning Re- search, 15:1929–1958, 2014. 1 Georg Still.Lectures on Parametric Optimization: An Introduction. Preprint, University of Twente, 2018. 16 Ke Sun, Zhanxing Zhu,...

work page 1929

[9] [9]

Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance.arXiv preprint arXiv:2304.11127, 2023

1 Ilya O Tolstikhin, Sylvain Gelly, Olivier Bous- quet, Carl-Johann Simon-Gabriel, and Bernhard Sch¨ olkopf. AdaGAN: Boosting Generative Mod- els.Advances in Neural Information Processing Sys- tems, 30, 2017. 6 Leslie G. Valiant. A Theory of the Learnable.Com- munications of the ACM, 27(11):1134–1142, 1984. 1 Vladimir Vapnik.The Nature of Statistical Lear...

work page arXiv 2017

[10] [10]

Multi-class AdaBoost.Statistics and Its Interface, 2:349–360, 2009

7 Ji Zhu, Hui Zou, Saharon Rosset, and Trevor Hastie. Multi-class AdaBoost.Statistics and Its Interface, 2:349–360, 2009. 2, 3 Checklist

work page 2009

[11] [11]

Yes (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm

For all models and algorithms presented, check if you include: (a) A clear description of the mathematical set- ting, assumptions, algorithm, and/or model. Yes (b) An analysis of the properties and complexity (time, space, sample size) of any algorithm. Not Applicable (c) (Optional) Anonymized source code, with specification of all dependencies, including...

work page

[12] [12]

Yes (b) Complete proofs of all theoretical results

For any theoretical claim, check if you include: (a) Statements of the full set of assumptions of all theoretical results. Yes (b) Complete proofs of all theoretical results. Yes (c) Clear explanations of any assumptions. Yes

work page

[13] [13]

Yes (b) All the training details (e.g., data splits, hy- perparameters, how they were chosen)

For all figures and tables that present empirical results, check if you include: (a) The code, data, and instructions needed to reproduce the main experimental results (ei- ther in the supplemental material or as a URL). Yes (b) All the training details (e.g., data splits, hy- perparameters, how they were chosen). Yes (c) A clear definition of the specifi...

work page

[14] [14]

Yes (b) The license information of the assets, if ap- plicable

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include: (a) Citations of the creator If your work uses ex- isting assets. Yes (b) The license information of the assets, if ap- plicable. Yes (c) New assets either in the supplemental mate- rial or as a URL, if applicable. Yes (d) Information about ...

work page

[15] [15]

Not Applicable (b) Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable

If you used crowdsourcing or conducted research with human subjects, check if you include: (a) The full text of instructions given to partici- pants and screenshots. Not Applicable (b) Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable. Not Applica- ble (c) The estimated hourly wage paid to ...

work page

[16] [16]

We show that PENEX is Fisher consistent, conditionally on anyxand for anyρ >0

work page

[17] [17]

""Exponential loss on the true-class score fy(x)

We show that PENEX is also unconditionally Fisher consistent. Step 1The population-level equivalent of (3), conditionally onx, is Ly|x PENEX(f(x)) :=E y|x h exp n −αf (y)(x) oi +ρ KX j=1 exp n f(j)(x) o .(11) We note that (11) is strictly convex inf(x) and differentiable. Hence, the unique minimum of (11), calledf ∗(x), is characterized by its gradient be...

work page 2018