{"id":"57056c98-b23c-443f-a3a4-0705dc8e504d","arxiv_id":"2608.11140","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A large simulation study finds that GiViTI, McCullagh, Osius-Rojek, le Cessie, and Stute-Zhu tests perform best for sparse-data logistic regression, and that visual calibration checks remain necessary.","lead":"A statistics thesis compares about 30 ways to test whether a logistic regression model fits sparse data. It finds five tests that best balance catching bad models with avoiding false alarms, and says visual calibration plots should still be used.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multiplicity gap: the five tests are each calibrated at alpha=0.05, but the paper recommends combining them without multiple-testing control, so the 'no false alarms' property is not established for the recommended multi-test workflow.","rationale":"The reader's weakest assumption is external validity: the ranking may not generalize beyond the Hosmer-style IID, low-dimensional, omitted-term scenarios. That concern is real and is partially conceded by the manuscript itself in Section 5.1.1, where the author notes the reliance on IID data, low-dimensional classical logistic regression, and limitations in detecting certain misspecifications. I do not dispute that concern, but I think the single most load-bearing issue is different and more immediate: the recommended workflow itself, as stated, lacks Type I error control. The abstract and Section 6.4 advise combining several powerful statistical tests with visual inspection, yet every simulation result and every 'size' statement is per-test at alpha=0.05. No combination rule, multiplicity correction, or familywise error rate is reported. Consequently, the central promise that these tests 'do not raise false alarms on good models' has not been demonstrated for the multi-test procedure that the paper actually recommends. A researcher who applies the five tests together and flags the model whenever any test is significant will reject correct models far more often than 5% in many settings. This is not a question of extending to new data regimes; it is a gap in the paper's own inferential claim. The concrete test I propose would settle it by directly estimating the familywise rejection rate from the existing simulation framework, using the union-of-tests rule that the phrase 'combination of several tests' naturally suggests. If the manuscript instead intends the tests to be used as exploratory complements with no formal combined decision, then the headline should be reworded to avoid implying familywise reliability. Either way, the verdict remains CONDITIONAL: the central claim needs an explicit, validated combination rule or a narrower statement of what 'not raising false alarms' means for the recommended multi-test workflow.","tokens_in":52232,"tokens_out":9344,"duration_ms":101676,"concrete_test":"Reanalyze the stored simulation output, or rerun the null scenarios of Section 5.2 with 10,000 replications, and compute the empirical familywise rejection rate when the decision rule is 'reject if any of GiViTI internal, McCullagh, Osius-Rojek, le Cessie, or Stute-Zhu rejects at alpha=0.05.' If this rate substantially exceeds 0.05 (e.g., is above 0.10) in any of the five null scenarios, the recommendation to combine these tests conflicts with the claimed Type I error balance; a corrected rule (e.g., Bonferroni or a pre-specified composite) would then be required before the headline guidance is issued.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recommendation is to use several powerful tests together (Section 6.4), and the abstract promises that the five identified tests balance power against 'not raising false alarms on good models.' But the simulation (Sections 5.2 and 5.8) evaluates each test individually at alpha=0.05; it never defines a combined decision rule or adjusts for multiplicity. If an applied researcher follows the recommendation and treats rejection by any of the five tests as evidence of poor fit, the chance of at least one false rejection under a correct model is 1-(1-0.05)^5, approximately 0.226 under independence, and can differ under the true correlations among the tests. The manuscript contains no Bonferroni, Benjamini-Hochberg, or any familywise control discussion. Thus the claimed balance between power and Type I error is per-test only; it does not carry over to the recommended 'combination of several powerful statistical tests' unless a combination rule is specified and validated. This concern is internal to the paper's own simulation evidence, independent of the acknowledged external-validity limits in Section 5.1.1.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This M.Sc. thesis manuscript compares approximately 30 goodness-of-fit and calibration tests for binary logistic regression under sparse data, following the simulation design of Hosmer et al. (1997). For sample sizes 200 to 5,000 and 10,000 replications per scenario, it estimates empirical Type I error under five covariate distributions and empirical power for omitted quadratic and omitted interaction misspecifications. It identifies five tests—GiViTi calibration, McCullagh, Osius–Rojek, le Cessie, and Stute–Zhu—as empirically powerful while maintaining correct Type I error, and it concludes that formal tests should be combined with visual calibration inspection. A real-data application to the Low Birth Weight dataset illustrates that many tests give conflicting conclusions.","tokens_in":52517,"tokens_out":5663,"duration_ms":57144,"significance":"If the empirical ranking is reliable, the paper provides a practically useful resource for applied statisticians choosing goodness-of-fit tests for logistic regression with continuous covariates. The study has notable strengths: the simulation uses an established benchmark, the number of replications is large, the exclusion of non-applicable or degenerate tests is transparent, the appendices include detailed algorithms and code, and the real-data example adds a concrete illustration. However, the paper is an empirical comparison, not a methodological advance, and its central recommendation depends on a statistical property—the behavior of the recommended multi-test workflow—that the simulation never actually evaluates. The external-validity limits of the Hosmer et al. scenario (IID, low-dimensional, two misspecification types) are acknowledged in Section 5.1.1, so those limits are not the main concern.","major_comments":[{"comment":"The recommendation to use 'a combination of several powerful statistical tests' (Section 6.4) is not supported by the simulation evidence, which evaluates each test at alpha=0.05 individually (Sections 5.2 and 5.8). The abstract's claim that the five tests balance power with 'not raising false alarms on good models' is per-test; if a practitioner treats rejection by any of the five as evidence of poor fit, the chance of at least one false rejection under a correct model is approximately 1-(0.95)^5 = 0.226 under independence, which is far above the nominal 5%. No Bonferroni, Benjamini-Hochberg, or any other familywise control is discussed, and no combined decision rule is defined or validated. Please either specify and validate a multi-test decision rule with multiplicity adjustment, or restrict the recommendation to per-test screening and state that the visual calibration step, rather than the data-driven combination of p-values, is what synthesizes the evidence.","section":"6.4 / Abstract"},{"comment":"Rejection rates are reported as raw proportions from 10,000 replications without Monte Carlo standard errors or confidence intervals. For a true rejection rate of 0.05, the binomial standard error is sqrt(0.05*0.95/10000) = 0.0022, so observed rates of 4.8% and 5.5% are statistically indistinguishable. Several passages in Section 5.8 describe tests as showing 'excellent control' or being 'robust' based on small absolute differences in these heatmaps. Reporting Monte Carlo standard errors or Wilson confidence intervals would let the reader judge which differences are meaningful and would make the ranking of 'best' tests more credible.","section":"5.4 / Figures 5.2–5.4"},{"comment":"The classification of the 'Hosmer Bootstrap' procedure as a liberal test appears inconsistent with the method described in Section 4.3.8, where Lai and Liu's standardized-power procedure is designed to prevent the ordinary Hosmer-Lemeshow test from rejecting large but practically well-fitting models. Table 5.3 records the Lai-Liu decision as a 0/1 p-value, which is not an ordinary p-value, and Section 5.7 then groups this test with Pearson and deviance as liberal. This suggests a possible implementation or interpretation mismatch. Please clarify how the test was implemented, what threshold was used, and whether the exclusion reflects a property of the original method or of the current implementation.","section":"5.7 / 4.3.8 / Table 5.3"}],"minor_comments":[{"comment":"There are numerous typos and grammatical errors throughout, including 'emperical power', 'thoes', 'assypmtotically', and 'model access requires'. A careful proofread is needed before publication.","section":"Abstract and Chapter 5"},{"comment":"The naming of the Copas unweighted sum-of-squares test is inconsistent: it is called 'Copas USS test with Osius and Rojek normal approximation' in Section 3.16.3 but appears as 'le Cessie-van Houwelingen-Copas-Hosmer unweighted sum of squares test' in the text. Please standardize the terminology.","section":"3.16.3 / 4.2.3"},{"comment":"Section 5.3.3 states that link-function misspecification is beyond the scope of the research, yet the Stukel score test is listed in Table 5.3 and described in detail in Section 3.13. Clarify whether the Stukel test is included in the main simulation or only in the excluded-test analysis.","section":"5.3.3 / Table 5.3"},{"comment":"The real-data figures plot p-values for many tests on a single axis, and the range of p-values makes small values hard to read. Consider using a log-scale or separate panels, and include the sample size and number of events in the figure captions.","section":"5.9 / Figures 5.9-5.12"},{"comment":"Table 5.2 reproduces values from Hosmer et al. (1997), but the source is not fully identified in the table footnote. Please add the original source and confirm permission/reproduction rights.","section":"Table 5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is written as a thesis chapter and would benefit from substantial editing before journal submission, but the simulation comparison is a useful practical contribution. The main technical issue is the gap between the per-test claims and the multi-test recommendation; this is fixable by either defining a valid combined rule or softening the conclusion. The paper is more of an applied comparison than a methodological novelty, so its fit depends on whether the journal accepts empirical benchmarking studies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on logistic regression diagnostics. This M.Sc. thesis posted on arXiv benchmarks about 30 goodness-of-fit tests and calibration procedures under the Hosmer et al. (1997) simulation framework. The genuinely useful thing is breadth plus transparency: it brings modern methods (GiViTI, eHL, BAGofT, Stute-Zhu, projection test) into one sparse-data comparison with 10,000 replications, and it documents why some tests are excluded. The five tests it highlights—GiViTI internal, McCullagh, Osius-Rojek, le Cessie, Stute-Zhu—are plausible winners in these scenarios: they hold size near the nominal level and have decent power against omitted quadratic and interaction terms. The recommendation to complement formal tests with calibration plots is sensible and matches practice. The math is mostly exposition of known tests rather than new derivation, which is appropriate for a benchmark; the citation pattern is honest and connects to the foundational and modern literature.\n\nThe main soft spot is the multiplicity gap the stress test flagged. The abstract and conclusion recommend a combination of several powerful tests, but the simulation evaluates each test individually at alpha = 0.05. If a researcher follows the advice and treats rejection by any of the five as evidence of poor fit, the false-alarm rate under an independent correct model is about 23%, not 5%. The paper provides no combination rule and no multiple-testing adjustment. The per-test claims are fine; the workflow claim is not established. A referee should ask for either a pre-specified combined decision rule or a clear statement that per-test p-values are reported without familywise control.\n\nTwo smaller issues. There are no Monte Carlo error bars; with 10,000 replications the binomial standard error near 5% is about 0.002, so rankings of close tests may be noisy. And the post hoc exclusion of liberal and zero-power tests is transparent but partly driven by practical constraints (e.g., the projection test's cost), so 'best five' is conditional on what was feasible. The real-data application uses a single small dataset, which limits what it adds beyond the simulations.\n\nBottom line: a useful empirical benchmark, not a new method or theory. Applied statisticians with continuous covariates get a reasonable evidence base, provided they don't read the 'combination' advice as a validated familywise procedure. The paper deserves a serious referee; the core comparison is publishable after a revision fixing the multiplicity issue and adding error bars.","headline":"A broad, transparent benchmark with plausible winners, but the recommended multi-test workflow is missing multiple-testing control.","tokens_in":53000,"tokens_out":4339,"would_cite":false,"duration_ms":42272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62J12","62F40"],"pacs":[],"model":"deepseek-v4-flash","headline":"Five tests stay reliable for sparse-data logistic regression","keywords":["goodness-of-fit tests","logistic regression","sparse data","calibration","Type I error","statistical power","simulation study","calibration plots"],"falsifier":"Re-run the same 30 tests with a misspecified link function (e.g., Stukel's generalized logistic with nonzero shape parameters) or with clustered outcomes: if any of the five champion tests shows a grossly inflated Type I error, or if Farrington suddenly acquires power, the empirical ranking is scenario-dependent rather than general.","tokens_in":52041,"feed_emoji":"📊","tokens_out":5655,"duration_ms":51196,"temperature":0.7,"pith_summary":"This thesis asks which goodness-of-fit tests for binary logistic regression still work when data are sparse, meaning each covariate pattern contains only one observation. It simulates about 30 classical and machine-learning calibration tests under the Hosmer et al. (1997) benchmark scenarios, with sample sizes from 200 to 5,000 and 10,000 replications per scenario. At a fixed significance level, the GiViTI calibration test, McCullagh's standardized Pearson test, Osius-Rojek, le Cessie-van Houwelingen, and Stute-Zhu have both high empirical power and correct Type I error, whereas Pearson, deviance, Farrington, and several variants are either liberal, powerless, or inapplicable. The thesis concludes that no single formal test is enough and that visual calibration plots remain a necessary complement.","feed_headline":"Five tests stay reliable for sparse-data logistic regression","feed_subtitle":"A 30-test simulation finds GiViTI, McCullagh, Osius-Rojek, le Cessie, and Stute-Zhu balance power against false alarms.","key_machinery":"The simulation template from Hosmer et al. (1997) is the centerpiece: covariates are drawn from uniform, normal, chi-square, and multi-independent distributions; the null distribution of each test is examined under correct specification; and power is assessed under two controlled misspecifications, an omitted quadratic term and an omitted interaction, whose true coefficients are fixed by solving small systems of logit equations based on chosen fixed points. Each test is evaluated by its empirical Type I error at α=0.05 and empirical power over 10,000 replications, with the 'good' tests required to do well on both criteria. The central identity tying the thesis together is the definition of the five best tests working as standardized or calibration-based statistics rather than as raw chi-square counts.","core_discovery":"The paper establishes an empirical ranking: when logistic regression is fit to sparse data, the tests that best balance size and power are the GiViTI calibration test (internal validation version), McCullagh's conditional-moment standardization of the Pearson statistic, Osius-Rojek's normal approximation, le Cessie-van Houwelingen's smoothed residual score test, and the Stute-Zhu cumulative-residual process test. These five maintain rejection rates close to the nominal 5% under correct specification and detect omitted quadratic and interaction terms that the classical tests miss. The classical Pearson and deviance tests are liberal in sparse settings, Farrington's test has essentially zero power, and the Unreliability (U) index and Spiegelhalter's z-test are insensitive to non-linear misspecification. Consequently, the thesis argues model assessment requires a battery of powerful formal tests together with visual inspection of calibration.","pith_inferences":["The ranking is established within the Hosmer benchmark's IID, low-dimensional scenarios; it may not transfer to clustered, high-dimensional, or link-misspecified settings, where a different set of tests could win.","A practical battery could be formed from the five best tests, with a decision rule based on agreement across tests rather than any single p-value; the thesis's own results support this but stop short of defining such a rule.","Test developers now have a concrete benchmark: a new GOF test should beat or match the five best on both Type I error and power across these exact scenarios.","Because the Unreliability index and Spiegelhalter's z-test are insensitive to non-linear misspecification, they are best used only as secondary checks of average calibration, not as omnibus GOF tests."],"forward_implications":["Applied researchers should prefer the GiViTI, McCullagh, Osius-Rojek, le Cessie-van Houwelingen, and Stute-Zhu tests when fitting logistic models with continuous predictors.","Classical Pearson, deviance, and related chi-square-based tests should not be used alone for sparse data, since they reject good models too often or fail to detect bad ones.","Calibration plots should be used routinely alongside formal tests, because formal methods alone miss patterns the eye can catch.","For moderate-to-large samples up to 5,000, the large-sample HL modification behaves similarly to the traditional HL test, so the added complexity is only justified for very large datasets."],"supporting_citations":[{"why":"Supplies the simulation design and benchmark scenarios the thesis adopts for all comparisons.","marker":"D. Hosmer, T. Hosmer, et al. 1997"},{"why":"Defines the original Hosmer-Lemeshow test that the thesis compares against and whose variants it studies.","marker":"Hosmer Jr and Lemeshow 1980"},{"why":"Provides the conditional-moment standardization that yields the McCullagh test, one of the five best.","marker":"McCullagh 1985"},{"why":"Introduces the normal approximation to the Pearson statistic underlying the Osius-Rojek test.","marker":"Osius and Rojek 1992"},{"why":"Defines the le Cessie-van Houwelingen smoothed residual score test, among the five best.","marker":"Cessie and Houwelingen 1995"},{"why":"Defines the cumulative-residual process test with model-based bootstrap, among the five best.","marker":"Stute and Zhu 2002"},{"why":"Defines the GiViTI internal calibration test and its null distribution, among the five best.","marker":"Nattino, Finazzi, et al. 2016"},{"why":"Shows why the chi-square approximation fails in sparse data and supplies a competing standardization that the simulations find powerless.","marker":"Farrington 1996"}],"fun_headline_variants":["Sparse logistic regression: five GOF tests balance size and power","Five logistic GOF tests keep their power on sparse data","Chi-square fails for sparse logistic, but these five don't","GiViTI, McCullagh, Osius-Rojek, le Cessie, Stute-Zhu lead sparse GOF","For sparse logistic fits, rely on five tests beyond chi-square"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking rests on the assumption that the Hosmer et al. simulation scenarios, with IID data, four covariate distributions, and misspecification limited to an omitted quadratic or interaction term, are representative enough of real sparse-data practice that a test winning there will also win in applications.","fun_headline_variants_meta":{"raw":{"variants":["Sparse logistic regression: five GOF tests balance size and power","Five logistic GOF tests keep their power on sparse data","Chi-square fails for sparse logistic, but these five don't","GiViTI, McCullagh, Osius-Rojek, le Cessie, Stute-Zhu lead sparse GOF","For sparse logistic fits, rely on five tests beyond chi-square"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001197,"raw_usage":{"total_tokens":4969,"prompt_tokens":1014,"completion_tokens":3955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":3855}},"tokens_in":630,"tokens_out":3955,"duration_ms":27517,"temperature":1.0,"reasoning_tokens":3855,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:21:59.377514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 30 tests with a misspecified link function (e.g., Stukel's generalized logistic with nonzero shape parameters) or with clustered outcomes: if any of the five champion tests shows a grossly inflated Type I error, or if Farrington suddenly acquires power, the empirical ranking is scenario-dependent rather than general.","supporting_citations":[{"cited_title":"and Finazzi, S","cited_arxiv_id":null,"evidence_quote":"Defines the GiViTI internal calibration test and its null distribution, among the five best."}],"review_version":1}