Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

Gradient Boosting Machine: A Survey

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Gradient boosting's many algorithms follow from one functional-gradient scheme.

desk verdict A survey that is useful as an index to the GBM literature but not as a mathematical reference, because the printed equations are demonstrably malformed at several load-bearing points. read the letter →

arxiv 1908.06951 v1 pith:JCOG4RDR submitted 2019-08-19 stat.ML cs.LG

Zhiyuan He , Danchen Lin , Thomas Lau , Mike Wu This is my paper · ORCID
classification stat.MLcs.LG MSC 62G0868T05
keywords gradientboostingfunctionaldescentAdaBoostLogitlossfunctionsrankingLambdaMARTsurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey argues that the many named gradient boosting algorithms are not separate inventions but instances of a single iterative scheme: minimize the expected loss by taking functional gradients and performing line search. It works through the derivations for AdaBoost, LogitBoost, least-squares boost, LAD regression, M-regression, two-class and multiclass logistic regression, and the ranking algorithm LambdaMART, and it presents the loss functions that drive each one. If the derivations are correct, the survey gives practitioners a single mathematical reference point: once a loss is chosen, the functional gradient and the stagewise update tell you how to construct the next model.

What carries the argument

The load-bearing identity is the functional-gradient formula $g_m(x) = E_y[\partial L(y,F(x))/\partial F(x) \mid x]$ evaluated at $F=F_{m-1}(x)$, combined with the stagewise greedy update $(\beta_m,\mathbf{a}_m) = \arg\min_{\beta,a} \sum_i L(y_i, F_{m-1}(x_i)+\beta h(x_i;a))$, where $h(x;a)$ is a weak base learner such as a regression tree. This pair carries the argument because it converts an arbitrary loss function into a numerical updating procedure; each algorithm in the survey is characterized by which loss it plugs in and what the resulting gradient, line search, or node update looks like.

What would settle it

Take a loss with an unbounded or non-integrable derivative, such as $L(y,F)=|y-F|^\alpha$ with $0<\alpha<1$, and a distribution that puts mass at $y=F$; compute $\partial E[L| x]/\partial F$ and $E[\partial L/\partial F|x]$ at that point. The survey's claim that these can be switched interchangeably implies they are equal, so any distribution where they differ refutes the unrestricted formula.

Watch

Extended reading notes

Core claim

In the paper's own terms, every boosting algorithm it reviews is a solution to the same functional optimization problem, $F^* = \arg\min_F E_{y,x} L(y,F(x))$, solved by iterative back-fitting. The central mathematical object is the functional gradient $g_m(x) = E_y[\partial L(y,F(x))/\partial F(x) \mid x]$ evaluated at $F=F_{m-1}(x)$, followed by a line search for the step size $\rho_m$. From this template, AdaBoost, LogitBoost, least-squares boost, LAD, M-regression, two-class and multiclass logistic regression, and LambdaMART each follow by choosing a loss and computing the resulting gradient and node updates. If the derivations are correct, the paper is an accurate mathematical reference for the GBM family.

Load-bearing premise

Section 2.1 assumes the derivative of the expected loss and the conditional expectation of the derivative can be switched interchangeably, with no regularity condition such as dominated convergence stated; if that swap fails, the simplified gradient formula and the greedy updates built on it are not justified.

Editorial extensions

If this is right

  • Implementing a new GBM variant reduces to choosing a loss and deriving its functional gradient and line search; the back-fitting machinery stays fixed.
  • The formulas in Section 4 make each algorithm's update concrete: LAD uses region medians, M-regression uses Huber-based median corrections, and the logistic variants use weighted sums of pseudo-responses.
  • The ranking objective and LambdaMART show that the same scheme extends to ranking by plugging in a ranking loss and using its lambda gradient.
  • Because AdaBoost's exponential loss is treated as overfitting-prone, the surveyed replacement with normalized sigmoid cost fits the same gradient-descent framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not pursue: the functional-gradient formula's validity depends on interchanging a derivative with an expectation, so one could check which standard loss families satisfy dominated convergence and thereby extend the framework to heavy-tailed or non-smooth losses.
  • An implicit consequence of the ranking objective is that the trade-off weight $w$ between preference pairs and labeled targets should control the bias-variance balance in ranking, analogous to the paper's own bias-variance discussion for weighted sampling.
  • The survey's framing suggests that any new ranking loss can be dropped into LambdaMART by deriving a lambda gradient from it; comparing NDCG-based lambdas with lambdas from other ranking metrics would be a direct test of how general the template is.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper is a survey of gradient boosting machines. It outlines the optimization framework for parametric and non-parametric models, lists common loss functions for continuous and categorical responses, presents several boosting variants (AdaBoost, Arc-x4, least-squares boost, LogitBoost, LAD regression, M-regression, two-class and multiclass logistic regression), and discusses ranking applications including LambdaMART. The authors state in the introduction that the survey concentrates on the mathematical derivations of gradient boosting algorithms, and the abstract promises that the mathematical frameworks are illustrated in detail.

Significance. The survey addresses a useful reference goal: collecting GBM variants and their loss functions in one place, with pointers to canonical literature such as Freund and Schapire (1997), Friedman (2001, 2002), Friedman et al. (2000), and Natekin and Knoll (2013). Its breadth of coverage is a genuine strength, and a careful corrected version could serve as a convenient entry point for practitioners and students. However, the value of a survey of this kind rests entirely on the accuracy of the exposition, and as printed the manuscript does not deliver the promised detailed mathematical derivations because several key displayed equations are malformed, self-referential, or missing necessary regularity conditions. There are no new results, machine-checked proofs, or empirical evaluations to offset these issues, so the work must be judged on the reliability of its mathematical presentation.

major comments (5)
  1. [Section 2.1] The displayed optimization problem for the parametric case, P* = argmin_p Phi(P)Phi(P) = E_{y,x} L(y,F(x;P))F*(x) = F(x;P*), is not well-formed: an argmin is equated to a product of Phi(P) with itself, then to an expectation multiplied by F*(x), and finally to F(x;P*). The definition P* = sum_{m=0}^M pm is also inconsistent with the preceding notation for P as the collection {beta_m, a_m}_1^M. This passage is the foundational optimization statement of the survey, so the malformed equation directly undermines the paper's central claim of presenting mathematical frameworks in detail.
  2. [Section 2.1] The statement that "the differentiation and integration of the gradient function can be switched interchangeably" is made without any regularity condition such as dominated convergence, and the resulting formula gm(x) = E_y[delta Phi(F(x))/delta F(x) | x] retains Phi inside the expectation. A correct interchange step would produce gm(x) = E_y[partial L(y,F)/partial F | x] evaluated at F = F_{m-1}(x). As printed, the derivation is not valid, and the simplified gradient formula that motivates the stagewise greedy updates is therefore not justified.
  3. [Section 4 (Two Class Logistic Regression)] The line-search update defines tilde y_i = 2 tilde y_i / (1 + exp(2 y_i F_{m-1}(x_i))), which is self-referential and cannot be evaluated. The intended Newton-step residual should be expressed in terms of the original response y_i and the current margin, for example tilde y_i = 2 y_i / (1 + exp(2 y_i F_{m-1}(x_i))) in the convention of Friedman (2001). The multiclass logistic regression update has the same defect. Because these equations are the central mathematical content of the logistic-regression sections, the survey does not currently provide a usable derivation.
  4. [Section 4 (LAD Regression)] The LAD update F_m(x) = F_{m-1}(x) + sum_{j-1} gamma 1(x in R_jm) is malformed: the summation is written as "j-1" instead of "j=1 to J", and the term should be gamma_jm 1(x in R_jm). The missing index makes the region-based update unintelligible. This is not a mere typographical nuisance; it obscures the precise algorithm being presented and further undercuts the survey's claim to provide detailed mathematical derivations.
  5. [Section 5] In the ranking objective, the constraint is stated as h(x_i) <= h(y_i) + tau "if x_i is ranked higher than y_i," while the objective contains the hinge term (max{0, h(y_i) - h(x_i) + tau})^2; the direction of the inequality appears inconsistent with the hinge term. In the LambdaMART gradient, gamma_{i,j} := S_{ij} |Delta NDCG delta C_{ij}/delta o_{ij}| mixes the absolute value of a product of Delta NDCG and the derivative of the cost, which is not the standard LambdaMART formulation and is not explained. This section therefore does not provide the detailed ranking derivation promised by the abstract.
minor comments (5)
  1. [Section 2.1] The notation Phi is used both as the objective functional and inside the expectation in the simplified gradient formula; introducing a separate symbol for the loss, such as L, would avoid confusion.
  2. [Section 3.1] In the quantile loss function, the conditions are written using y - f <= 0 and y - f > 0; defining the residual r = y - f throughout would make the piecewise definition clearer.
  3. [Section 4 (LogitBoost)] The stagewise update F_{i,k} = F_{i,k} + v (K-1)/K (f_{i,k} - (1/K) sum_{k=0}^{K-1} f_{i,k}) uses f_{i,k} both as the objective function and as the weak-learner output; please clarify the notation and define the summation index explicitly.
  4. [Section 4 (M-Regression)] The formula for gamma_jm is dense and leaves N_{jm} and delta_m undefined; these quantities should be defined or the reader should be referred to the precise equation in Friedman (2002).
  5. [Section 5] The definition Ni := ni sum_{j=1}^T (2^{r(j)} - 1)/log(1 + j) contains an unexplained ni before the summation; please define this normalization factor.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: expository survey with external attributions; malformed displays are correctness issues, not circular reductions.

full rationale

This is a survey with no novel predictions and no fitted parameters, and the central claim is expository accuracy rather than a derived result. The gradient-boosting framework is not defined in terms of the paper's own output; it reproduces standard stagewise optimization attributed to external literature (Freund and Schapire 1997; Friedman 2001, 2002; Mason et al. 1999, 2000; Wu et al. 2008), and none of the cited works are by the present authors, so no self-citation chain is load-bearing. The only apparently self-referential object is the display in Section 4 where ~tilde y_i appears on both sides of the two-class logistic-regression update; this is a typo in the pseudo-response formula and a correctness defect, not a circular reduction, because the survey does not use that equation to derive its own conclusion. Similarly, the unstated dominated-convergence condition in Section 2.1 is a rigor gap, not a circularity. No step satisfies the required quote-and-reduction standard for circularity, so the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey makes no new quantitative claim, so the ledger of fitted parameters is empty. Its expository content rests on standard bibliographic sources and on a few unstated mathematical preconditions, mainly in Section 2: differentiability of the loss, interchange of derivative and expectation, and convergence of greedy stagewise fitting. No entities are invented.

assumptions (3)
  • domain assumption L(y,F) is differentiable with respect to F so that steepest descent in function space is defined.
    Section 2.1 computes derivatives of the expected loss with respect to F and uses them as descent directions for all subsequent algorithms.
  • domain assumption Differentiation and conditional expectation can be interchanged when computing gm(x).
    Section 2.1 asserts 'The differentiation and integration of the gradient function can be switched interchangeably' without stating dominated convergence or smoothness conditions.
  • domain assumption Stagewise additive updates converge to the minimizer F*(x).
    Section 2.2 replaces the global minimization with greedy stagewise fitting and states Fm(x)=Fm-1(x)+beta_m h without a convergence proof.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gradient Boosting Machine: A Survey." pith.science (2026). https://pith.science/paper/JCOG4RDR

@misc{pith2026190806951,
  author       = {Pith},
  title        = {Pith review of: Gradient Boosting Machine: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JCOG4RDR}},
  note         = {Machine review of arXiv:1908.06951}
}
read the original abstract

In this survey, we discuss several different types of gradient boosting algorithms and illustrate their mathematical frameworks in detail: 1. introduction of gradient boosting leads to 2. objective function optimization, 3. loss function estimations, and 4. model constructions. 5. application of boosting in ranking.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Small Dataset May Go a Long Way: Process Duration Prediction in Clinical Settings

    stat.AP 2025-08 conditional novelty 5.0 of 10

    Cluster averages of historical surgery durations match or beat much more complex ML models and outperform manual plans on a small single-hospital dataset.

  2. Enhancing Plagiarism Detection in Marathi with a Weighted Ensemble of TF-IDF and BERT Embeddings for Low-Resource Language Processing

    cs.CL 2025-01 conditional novelty 4.0 of 10

    A weighted ensemble of TF-IDF and BERT features reaches 82.04% accuracy on Marathi plagiarism detection, outperforming BERT-only (80.64%).

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · cited by 2 Pith papers

  1. [1]

    Breiman, L. (1996). Bias, Variance, and Arcing Classifiers. Statistics Department, University of California, Berkeley, CA, USA. Tech. Rep. 460

    work page 1996

  2. [2]

    Breiman, L. (1997). ARCING THE EDGE. Statistics Department, University of California, Berkeley, CA, USA. Tech. Rep. 486

    work page 1997

  3. [3]

    E., & Singer, Y

    Collins, M., Schapire, R. E., & Singer, Y. (2002). Logistic Regression, Adaboost amd Bregman Distances. Machine Learning 48 , 253--285

    work page 2002

  4. [4]

    R., & Hastie, T

    Elith, J., and Leathwick, J. R., & Hastie, T. (2008). A Working Guide to boosted Regression Rrees. Journal of Animal Ecology 77 , 802--813

    work page 2008

  5. [5]

    Freund, Y., and Schapire, R. E. (1997). A Decision Theoretic Generalization of Online Learning and An Application to Boosting. Journal of Computer and System Sciences 55 , 119--139

    work page 1997

  6. [6]

    Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics, 1189--1232

    work page 2001

  7. [7]

    Friedman, J. H. (2002). Stochastic Gradient Boosting. Computational Statistics & Data Analysis 38 , 367--378

    work page 2002

  8. [8]

    Friedman, J., Hastie, T., & Tibshirani, R. (2000). ADDITIVE LOGISTIC REGRESSION: A STATISTICAL VIEW OF BOOSTING. The Annals of Statistics 28 , 337--407

    work page 2000

Show all 18 references
  1. [9]

    Koenker, R., and Hallock, K. F. (2001). Quantile Regression Journal of Economic Perspectives 15 , 143--156

    2001

  2. [10]

    Li, P. (2012). Robust Logitboost and Adaptive Base Class (abc) Logitboost. arXiv preprint arXiv:1203.3491

    2012 arXiv

  3. [11]

    Mason, L., Baxter, J., Bartlett, P., & Frean, M. (1999). Boosting Algorithms as Gradient Descent in Function Space

    1999

  4. [12]

    Mason, L., Baxter, J., Bartlett, P., & Frean, M. (2000). Boosting Algorithms as Gradient Descent. Advances in Neural Information Processing Systems, 512--518

    2000

  5. [13]

    Natekin, A., and Knoll, A. (2013). Gradient Boosting Machines, A Tutorial. Frontiers in Neurorobotics 7 , 21

    2013

  6. [14]

    Schapire, R. E. (1990). The Strength of Weak Learnability. Machine Learning 5 , 197--227

    1990

  7. [15]

    D., & Zhou., J

    Sun, P., Reid, M. D., & Zhou., J. (2012). ASOS-LogitBoost: Adaptive One-Vs-One LogitBoost for Multi-Class Problem. arXiv preprint arXiv:1110.3907

    2012 arXiv

  8. [16]

    Wang, R. (2012). AdaBoost for Feature Selection, Classification and Its Relation with SVM, A Review. Physics Procedia 25 , 800--807

    2012

  9. [17]

    J., Svore, K

    Wu, Q., Burges, C. J., Svore, K. M., & Gao, J. (2008). Ranking, Boosting, and Model Adaptation

    2008

  10. [18]

    Zheng, Z., Zha, H., Zhang, T., Chapelle, O., Chen, K., & Sun, G. (2008). A General Boosting Method and Its Application to Learning Ranking Functions for Web Search. Advances in Neural Information Processing Systems, 1697--1704

    2008

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.