REVIEW 4 major objections 5 minor 42 references
On sparse logistic-regression data, a five-test panel is more reliable and more powerful than the standard Hosmer–Lemeshow test.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
In a large simulation study, five existing goodness-of-fit tests (McCullagh, Osius-Rojek, le Cessie-van Houwelingen, Stute-Zhu, GiViTI) control Type I error and beat Hosmer-Lemeshow in power, but a real-data example shows no single test reliably catches omitted interactions.
T0 review reviewed 2026-08-01 challenge →
load-bearing objection Useful, reproducible benchmark that makes a solid case against the HL default, but the recommendation hinges on unverified implementations in the authors' own package. the 4 major comments →
Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that under fully sparse data (J ≈ n covariate patterns), the usual textbook tests are unreliable, but a compact set of five procedures—the standardized-Pearson tests of McCullagh and Osius–Rojek, the kernel-smoothing test of le Cessie–van Houwelingen, the CUSUM bootstrap test of Stute–Zhu, and the GiViTI polynomial calibration test—delivers empirical Type I error near the nominal 5% across a wide range of covariate distributions and sample sizes while showing higher power than Hosmer–Lemeshow against omitted quadratic and interaction terms. The paper also claims that no single test is sufficient: in the low-birth-weight example, a misspecified model with omitted interact
What carries the argument
The central object is the benchmark itself, organized through a six-family taxonomy of goodness-of-fit and calibration algorithms: partition/grouping tests, standardized-Pearson tests, score/link tests, smoothing tests, calibration tests, and bootstrap tests. The comparison runs under a standard simulation design with five covariate distributions, two misspecifications at two severities, five sample sizes, and 10,000 replications per scenario, with every test applied to the same simulated dataset so all comparisons are paired. The key mechanism shared by the recommended tests is recovering a valid null distribution under sparsity: Osius–Rojek and McCullagh standardize the Pearson statistic b
Load-bearing premise
The whole comparison rests on the authors' own implementations of the roughly twenty-five tests being faithful to the published methods, especially the McCullagh variance formula and the GiViTI internal null distribution; if those implementations are wrong, every size and power estimate—and the apparent advantage of the recommended core—could shift.
What would settle it
Run the recommended five tests on a published example with a known correct logistic fit using an independent implementation (for instance, a mainstream statistics package), and check whether the p-values match those reported here. If the McCullagh or GiViTI p-values differ systematically from the published methods, the benchmark's rankings are not reliable. Alternatively, rerun the simulation with a different implementation and see whether the five-test core still beats Hosmer–Lemeshow on sparse data.
If this is right
- Practitioners should stop relying on a single Hosmer–Lemeshow p-value when checking sparse logistic fits; the recommended five-test panel matches or beats its size control while being consistently more powerful.
- Raw Pearson, deviance, the Hosmer–Lemeshow F-test, and the bootstrap Hosmer test should not be used on sparse continuous data because they reject correctly specified models far too often.
- Over-conservative tests like Spiegelhalter and eHL, and essentially powerless tests like Farrington and the unreliability index, are unsafe as standalone checks.
- A calibration (reliability) diagram with a nonparametric curve can expose structured misfit that omnibus p-values miss, and should accompany any panel of tests.
- Even the recommended core can fail on real data, so robust assessment requires triangulating several diagnostics rather than trusting any single test.
Where Pith is reading between the lines
- If the benchmark is right, the same ranking may not hold in high-dimensional or penalized logistic models, since the paper explicitly leaves link-function misspecification, clustered data, and penalized fits for future work; a natural extension is to test the recommended core under ridge or lasso fits.
- The finding that grouped-data tests caught the omitted interaction in the low-birth-weight example suggests a hybrid strategy: in near-sparse data with a few repeated covariate patterns, complementing the sparse-data core with a grouped statistic may be more sensitive than any fully sparse test alone.
- The GiViTI internal null distribution and the McCullagh variance formula used in the paper are nonstandard; an independent implementation check against published numeric examples would directly test whether the benchmark's ranking is an artifact of the software.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper provides a taxonomy and an empirical comparison of roughly 25 goodness-of-fit and calibration tests for binary logistic regression in the sparse-data regime (continuous covariates, J≈n). The authors implement the tests in a new R package, ebrahim.gof, and run a large simulation study (10,000 replications; five covariate distributions for Type I error; two misspecification types at slight and pronounced severity for power; n=200–5000). Based on the simulated size and power, they recommend a core of five tests—McCullagh, Osius–Rojek, le Cessie–van Houwelingen, Stute–Zhu, and GiViTI—as more powerful than Hosmer–Lemeshow while maintaining size. A low-birth-weight example illustrates that a single p-value can miss omitted interactions and motivates pairing tests with a calibration curve.
Significance. If the benchmark is trustworthy, the practical payoff is high: it gives applied users an evidence-based alternative to the default Hosmer–Lemeshow test and a compact panel to report. The study is carefully executed at the simulation level: seeds, 10,000 replications, paired comparisons, Monte-Carlo standard errors, an α=0.01 robustness check, and an archived code repository. The taxonomy is useful even if the rankings were to shift. The main risk is that all test implementations come from the authors' own package, with no independent verification of the two most nonstandard algorithms (McCullagh and GiViTI). The paper's central recommendation therefore cannot be accepted on the current evidence without implementation-level validation.
major comments (4)
- [Appendix A, Algorithm 3, Step 6] The conditional variance formula Var(X^2|βhat) = (1-p*/n)(2Σ(m_i-1)/m_i + RSS_u) reduces to (1-p*/n)RSS_u for pure binary data. This is not the textbook form of McCullagh (1985) and is load-bearing: it determines the standardization of every McCullagh p-value and therefore the test's size and power in all tables. The paper offers no check against the original derivation, published numerical examples, or an independent implementation. Please add verification (e.g., reproduce known conditional moments in a simple design or compare with a published implementation) before the core recommendation can be accepted.
- [Appendix A, Algorithm 6, Step 20] The internal-validation null CDF F_m (Nattino et al. 2016, Eq. 10) is not a standard chi-square and is not implemented or documented in the paper. The GiViTI p-value, and hence its inclusion in the core, depends on the numerical evaluation of F_m. Without stating how F_m is computed (quadrature, simulation, closed form) and validating it (e.g., against givitiR or Nattino's tables), the reported Type I error and power for GiViTI cannot be independently reproduced.
- [Sections 3.3 and 4.4] Power is simulated only for x~U(-3,3), while the Type I study spans five covariate distributions. The core-selection rule in Section 4.4 requires 'ranks among the strongest performers in the slight-misspecification regimes', but those rankings come from a single covariate distribution. Given that size behavior changes substantially across the distributions in Figure 1, power rankings may also change under skewed/sparse covariates. Either run power scenarios for at least one additional covariate distribution (e.g., χ²(4) or U(-6,6)) or restrict the recommendation to the U(-3,3) misspecification settings. This is load-bearing for the general claim in the abstract that the core 'delivers the best balance' across sparse data.
- [Section 4.4] The recommended core is selected after inspecting the same simulation results it is then recommended from, with no confirmatory split or pre-specified selection rule. The qualitative 'simple reading' criterion is not fully operationalized (e.g., exact size thresholds and power ranks are not given), and the apparent advantage of the core could partly reflect selection on the same random draws. A validation round on held-out scenarios or a fully specified decision rule with sensitivity analysis would strengthen the central recommendation, especially because the implementations are from a single new package.
minor comments (5)
- [Abstract / Section 3] The abstract says 'four misspecification scenarios'; this should be clarified as two misspecification types, each at two severities, to avoid seeming broader than the design.
- [Section 3.1] The phrase 'the default bandwidth holding ≈√n points for le Cessie' is vague. Specify the kernel, the bandwidth rule, and whether it is the implementation in ebrahim.gof or the original paper's rule.
- [Algorithm 2, Step 9] For ungrouped data A=0 and the denominator becomes sqrt(RSS). If RSS is zero or near zero (e.g., intercept-only or saturated designs), the Osius–Rojek statistic is unstable. Please discuss or exclude such degenerate cases.
- [Algorithm 5, Step 12] The bootstrap p-value uses the indicator 1{T* ≥ T_obs}; please state the tie-handling convention and report the Monte Carlo error associated with B=200 bootstrap replications per fit.
- [Table 4] The 'PR+GAM' p-value of 0.069 is described as borderline; consider stating explicitly that it is not significant at the 5% level, to avoid ambiguity in the narrative.
Circularity Check
No significant circularity: the benchmark is an empirical evaluation against simulated ground truth; implementation-fidelity concerns are correctness risks, not circular steps.
full rationale
This paper is an empirical simulation benchmark, not a derivation from first principles. The recommended core is a summary of simulated Type I error and power under the published Hosmer et al. (1997) framework, with data generated from known correct and misspecified models. The tests are external published algorithms re-implemented in the authors' package ebrahim.gof; no test parameter is fitted to the benchmark outcomes and no fitted value is renamed as a prediction. The acknowledged dependence on the authors' own implementation (notably McCullagh's conditional variance and GiViTI's internal null CDF) is a reproducibility and fidelity risk, but it is not circularity: the package is a re-implementation of prior external methods, and the simulations have independent ground truth. There are no self-citations used as load-bearing evidence, no imported uniqueness theorems, and no ansatz smuggled in via self-citation. The taxonomy is explicitly non-canonical and adopted for convenience. Implementation bugs would affect the conclusions, but that is a correctness concern, not a circular derivation.
Axiom & Free-Parameter Ledger
free parameters (5)
- Quadratic severity anchor Q =
0.01 (slight), 0.4 (pronounced)
- Interaction severity anchor I =
0.1 (slight), 0.7 (pronounced)
- GiViTI forward-selection threshold q =
0.95
- le Cessie bandwidth =
≈ sqrt(n) points
- BAGofT resamples nsim =
100
axioms (5)
- domain assumption The Hosmer et al. (1997) simulation framework's data-generating scenarios are representative of realistic sparse logistic regression settings.
- standard math The asymptotic reference distributions used by each test (e.g., chi-square with G−2 df for HL, standard normal for Osius–Rojek and McCullagh) are valid under the simulated null.
- domain assumption The implementation of each test in ebrahim.gof is faithful to its published definition.
- standard math The model-based bootstrap for Stute–Zhu, which resamples responses from the fitted model, is valid under H0.
- domain assumption The anchor-probability method of setting coefficients controls severity without inducing pathological separation.
Cite this review
Pith. "Pith review of Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data." pith.science (2026). https://pith.science/paper/CSKU2NYW
@misc{pith2026260716344,
author = {Pith},
title = {Pith review of: Benchmarking Goodness-of-Fit and Calibration Algorithms for Logistic Regression Classifiers: A Large-Scale Simulation Study under Sparse Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/CSKU2NYW}},
note = {Machine review of arXiv:2607.16344}
}
read the original abstract
Binary logistic regression is among the most widely used classification algorithms, yet a classifier is only trustworthy if its predicted probabilities are well calibrated. The classical checks -- the Pearson chi-square and deviance statistics -- break down precisely in the modern setting where predictors are continuous and the data are sparse (one covariate pattern per observation). Four decades of research have produced dozens of alternative goodness-of-fit and calibration algorithms, yet practitioners still default to the Hosmer-Lemeshow test because it ships with their software. This paper provides a unified taxonomy and a large-scale, reproducible simulation benchmark; more than twenty tests are implemented in the open-source R package ebrahim.gof. We evaluate them across five covariate distributions and four misspecification scenarios, with 10,000 replications each, measuring both Type I error and power. Several classical tests prove liberal, rejecting correct models far too often, while others have little power. A compact core -- McCullagh, Osius-Rojek, le Cessie-van Houwelingen, Stute-Zhu, and the GiViTI calibration test -- delivers the best balance of correct size and high power, and is consistently more powerful than the ubiquitous Hosmer-Lemeshow test. A low-birth-weight application reinforces the point: a model with omitted interactions slips past nearly every test, exposed only by pairing sensitive tests with a calibration (reliability) curve. We translate these findings into practical, evidence-based guidance for assessing logistic regression fit.
Figures
Reference graph
Works this paper leans on
-
[1]
Agresti.Categorical data analysis
A. Agresti.Categorical data analysis. John Wiley & Sons, Hoboken, NJ, 3rd edition, 2013
2013
-
[2]
D. T. Ailobhio and J. A. Ikughur. A review of some goodness-of-fit tests for logistic regression model.Asian Journal of Probability and Statistics, 26(7):75–85, 2024. doi: 10.9734/ajpas/2024/v26i7631
-
[3]
G. W. Brier. Verification of forecasts expressed in terms of probability.Monthly Weather Review, 78(1):1–3, 1950. doi: 10.1175/1520-0493(1950)078 ⟨0001:VOFEIT⟩2.0.CO
-
[4]
URL https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_ 078_0001_vofeit_2_0_co_2.xml
-
[5]
J. D. Canary, L. Blizzard, R. P. Barry, D. W. Hosmer, and S. J. Quinn. A comparison of the Hosmer–Lemeshow, Pigeon–Heyse, and Tsiatis goodness-of-fit tests for binary logistic regression under two grouping methods.Communications in Statistics-Simulation and Computation, 46(3):1871–1894, 2017
2017
-
[6]
J. B. Copas. Unweighted sum of squares test for proportions.Journal of the Royal Statistical Society: Series C (Applied Statistics), 38(1):71–80, 1989
1989
-
[7]
Farrington
C.P. Farrington. On assessing goodness of fit of generalized linear models to sparse data. Journal of the Royal Statistical Society, Series B, 58(2):349–360, 1996
1996
-
[8]
S. Finazzi, D. Poole, D. Luciani, P. E. Cogo, and G. Bertolini. Calibration belt for quality- of-care assessment based on dichotomous outcomes.PLoS ONE, 6(2):e16110, 2011. doi: 10.1371/journal.pone.0016110
-
[9]
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning, pages 1321–1330, 2017
2017
-
[10]
Harrell, Jr.Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis
Frank E. Harrell, Jr.Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. Springer, Cham, second edition,
-
[11]
D. Hosmer, T. Hosmer, S. le Cessie, and S. Lemeshow. A comparison of goodness-of-fit tests for the logistic regression model.Statistics in Medicine, 16(9):965–980, 1997. doi: 10.1002/(sici)1097-0258(19970515)16:9⟨965::aid-sim509⟩3.0.co;2-o
-
[12]
Henzi, M
A. Henzi, M. Puke, T. Dimitriadis, and J. Ziegel. A safe Hosmer-Lemeshow test.The New England Journal of Statistics in Data Science, 2(2):175–189, 2024. doi: 10.51387/ 23-NEJSDS56. 22
2024
-
[13]
D. W. Hosmer Jr and S. Lemeshow. Goodness of fit tests for the multiple logistic regression model.Communications in statistics-Theory and Methods, 9(10):1043–1069, 1980
1980
-
[14]
D. W. Hosmer, S. Lemeshow, and R. X. Sturdivant.Applied Logistic Regression. John Wiley & Sons, Hoboken, NJ, 3rd edition, 2013
2013
-
[15]
A simple test procedure in standardizing the power of hosmer–lemeshow test in large data sets.Journal of Statistical Computation and Simulation, 88(13):2533–2543,
Xin Lai and Liu Liu. A simple test procedure in standardizing the power of hosmer–lemeshow test in large data sets.Journal of Statistical Computation and Simulation, 88(13):2533–2543,
-
[16]
O. Kuss. Global goodness of fit testing in logistic regression with sparse data.Statistics in Medicine, 21:3789–3801, 2002
2002
-
[17]
le Cessie and J
S. le Cessie and J. C. van Houwelingen. Testing the fit of a regression model via score tests in random effects models.Biometrics, 51(2):600–614, 1995
1995
-
[18]
H. Liu, X. Li, F. Chen, W. H¨ ardle, and H. Liang. A comprehensive comparison of goodness- of-fit tests for logistic regression models.Statistics and Computing, 34(175), 2024. doi: 10.1007/s11222-024-10487-5
-
[19]
le Cessie and J
S. le Cessie and J. C. van Houwelingen. A goodness-of-fit test for binary regression models, based on smoothing methods.Biometrics, 47(4):1267–1282, 1991
1991
-
[20]
G. Nattino, S. Finazzi, and G. Bertolini. A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes. Statistics in Medicine, 33(14):2390–2407, 2014. doi: 10.1002/sim.6100
-
[21]
G. Nattino, S. Finazzi, and G. Bertolini. A new test and graphical tool to assess the goodness of fit of logistic regression models.Statistics in Medicine, 35(5):709–720, 2016. doi: 10.1002/sim.6744
-
[22]
McCullagh
P. McCullagh. On the asymptotic distribution of Pearson’s statistic in linear-exponential family models.International Statistical Review, 53(1):61–67, 1985
1985
-
[23]
Niculescu-Mizil and R
A. Niculescu-Mizil and R. Caruana. Predicting good probabilities with supervised learning. InProceedings of the 22nd International Conference on Machine Learning, pages 625–632, 2005
2005
-
[24]
E. Nygaard. A simulation study of goodness-of-fit tests for binary regression with applications to Norwegian intensive care registry data. Master’s thesis, Univer- sity of Bergen, Bergen, Norway, 2019. URL https://www.helse-bergen.no/ 4905f6/siteassets/seksjon/intensivregister/documents/forskning-studier/ masterstudier/nygaard-kalibrering-av-sapsii.pdf . ...
2019
-
[25]
Giovanni Nattino, Michael L. Pennell, and Stanley Lemeshow. Assessing the goodness of fit of logistic regression models in large samples: A modification of the Hosmer-Lemeshow test. Biometrics, 76(2):549–560, 2020. doi: 10.1111/biom.13249
-
[26]
Osius and D
G. Osius and D. Rojek. Normal goodness-of-fit tests for multinomial models with large degrees of freedom.Journal of the American Statistical Association, 87(420):1145–1152, 1992
1992
-
[27]
J. G. Pigeon and J. F. Heyse. An improved goodness of fit statistic for probability prediction models.Biometrical Journal, 41(1):71–82, 1999
1999
-
[28]
C. Orme. The calculation of the information matrix test for binary data models.The Manchester School, 56(4):370–376, 1988. 23
1988
-
[29]
J. Shang, T. J. Robinson, and S. S. Wulff. Goodness-of-fit tests in logistic regression with continuous covariates.Journal of Modern Applied Statistical Methods, 20(2), 2021. doi: 10.56801/Jmasm.V20.i2.3
-
[30]
D. J. Spiegelhalter. Probabilistic prediction in patient management and clinical trials. Statistics in Medicine, 5(5):421–433, 1986
1986
-
[31]
Pulkstenis and T
E. Pulkstenis and T. J. Robinson. Two goodness-of-fit tests for logistic regression models with continuous covariates.Statistics in Medicine, 21:79–93, 2002
2002
-
[32]
T. A. Stukel. Generalized logistic models.Journal of the American Statistical Association, 83(402):426–431, 1988
1988
-
[33]
Stute and L.-X
W. Stute and L.-X. Zhu. Model checks for generalized linear models.Scandinavian Journal of Statistics, 29(3):535–545, 2002
2002
-
[34]
E. W. Steyerberg, A. J. Vickers, N. R. Cook, T. Gerds, M. Gonen, N. Obuchowski, M. J. Pencina, and M. W. Kattan. Assessing the performance of prediction models: A framework for some traditional and novel measures.Epidemiology, 21(1):128–138, 2010. doi: 10.1097/EDE.0b013e3181c30fb2
-
[35]
Van Calster, D
B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, and E. W. Steyerberg. Calibration: the achilles heel of predictive analytics.BMC Medicine, 17(1):230, 2019
2019
-
[36]
H. White. Maximum likelihood estimation of misspecified models.Econometrica, 50(1): 1–25, 1982
1982
-
[37]
A. A. Tsiatis. A note on a goodness-of-fit test for the logistic regression model.Biometrika, 67:250–251, 1980
1980
-
[38]
X. J. Xie, J. Pendergast, and W. Clarke. Increasing the power: A practical approach to goodness-of-fit test for logistic regression models with continuous predictors.Computational Statistics & Data Analysis, 52:2703–2713, 2008
2008
-
[39]
Jiawei Zhang, Jie Ding, and Yuhong Yang. Is a classification procedure good enough? A goodness-of-fit assessment tool for classification learning.Journal of the American Statistical Association, 118(542):1115–1125, 2023. doi: 10.1080/01621459.2021.1979010. 24
arXiv 2023
-
[40]
F. A. G. Windmeijer. The asymptotic distribution of the sum of weighted squared residuals in binary choice models.Statistica Neerlandica, 44:69–78, 1990
1990
-
[2015]
doi: 10.1007/978-3-319-19425-7
-
[2018]
doi: 10.1080/00949655.2018.1467912
arXiv 2018
This paper was first reviewed by deepseek-v4-flash on August 1, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.